Methods and apparatuses for forming a three-dimensional volumetric model of a subject&#39;s teeth

ABSTRACT

Methods and apparatuses for generating a model of a subject&#39;s teeth. Described herein are intraoral scanning methods and apparatuses for generating a three-dimensional model of a subject&#39;s intraoral region (e.g., teeth) including both surface features and internal features. These methods and apparatuses may be used for identifying and evaluating lesions, caries and cracks in the teeth. Any of these methods and apparatuses may use minimum scattering coefficients and/or segmentation to form a volumetric model of the teeth.

CROSS REFERENCE TO RELATED APPLICATIONS

This patent application is a continuation-in-part of U.S. patentapplication Ser. No. 15/662,250, titled “METHODS AND APPARATUSES FORFORMING A THREE-DIMENSIONAL VOLUMETRIC MODEL OF A SUBJECT'S TEETH,”filed Jul. 27, 2017, now U.S. Patent Application Publication No.2018/0028064, which claims priority to each of: U.S. Provisional PatentApplication No. 62/367,607, titled “INTRAORAL SCANNER WITH DENTALDIAGNOSTICS CAPABILITIES,” and filed on Jul. 27, 2016; U.S. ProvisionalPatent Application No. 62/477,387, titled “INTRAORAL SCANNER WITH DENTALDIAGNOSTICS CAPABILITIES,” filed on Mar. 27, 2017; and U.S. ProvisionalPatent Application No. 62/517,467, titled “MINIMAL VALUE LIFTING TO FORMA VOLUMETRIC MODEL OF AN OBJECT,” filed on Jun. 9, 2017. Each of theseis herein incorporated by reference in its entirety.

This patent application is also a continuation-in-part of U.S. patentapplication Ser. No. 16/258,516, titled “DIAGNOSTIC INTRAORAL SCANNING,”filed Jan. 25, 2019, which claims priority to U.S. Provisional PatentApplication No. 62/622,798, titled “DIAGNOSTIC INTRAORAL SCANNERS,”filed on Jan. 26, 2018, and U.S. Provisional Patent Application No.62/758,503, titled “DIAGNOSTIC INTRAORAL SCANNERS,” and filed Nov. 9,2018, each of which is herein incorporated by reference in its entirety.

INCORPORATION BY REFERENCE

All publications and patent applications mentioned in this specificationare herein incorporated by reference in their entirety to the sameextent as if each individual publication or patent application wasspecifically and individually indicated to be incorporated by reference.

FIELD

The methods and apparatuses described herein may relate to opticalscanners, and particularly for generating three-dimensionalrepresentations of objects. In particular, described herein are methodsand apparatuses that may be useful in scanning, including 3D scanning,and analyzing the intraoral cavity for diagnosis, treatment,longitudinal tracking, tooth measurement, and detection of dental cariesand cracks. These methods and apparatuses may generate volumetric modelsof the internal structure of the teeth, and/or may include colorscanning.

BACKGROUND

Many dental and orthodontic procedures can benefit from accuratethree-dimensional (3D) descriptions of a patient's dentation andintraoral cavity. In particular, it would be helpful to provide athree-dimensional description of both the surface, and internalstructures of the teeth, including the enamel and dentin, as well ascaries and the general internal composition of the tooth volume.Although purely surface representations of the 3D surfaces of teeth haveproven extremely useful in the design and fabrication of dentalprostheses (e.g., crowns or bridges), and treatment plans, the abilityto image internal structures including the development of caries andcracks in the enamel and underlying dentin, would be tremendouslyuseful, particularly in conjunction with a surface topographicalmapping.

Historically, ionizing radiation (e.g., X-rays) have been used to imageinto the teeth. For example, X-Ray bitewing radiograms are often used toprovide non-quantitative images into the teeth. However, in addition tothe risk of ionizing radiation, such images are typically limited intheir ability to show features and may involve a lengthy and expensiveprocedure to take. Some intraoral features such as soft tissues, plaqueand soft calculus may not be easily visualized via x-ray because oftheir low density. Other techniques, such as cone beam computedtomography (CBCT) may provide tomographic images, but still requireionizing radiation.

Thus, it would be beneficial to provide methods and apparatuses,including devices and systems, such as intraoral scanning systems, thatmay be used to model a subject's tooth or teeth and include bothexternal (surface) and internal (within the enamel and dentin)structures and composition using non-ionizing radiation. The model ofthe subject's teeth may be a 3D volumetric model or a panoramic image.In particular, it would be helpful to provide methods and apparatusesthat may use a single apparatus to provide this capability. There is aneed for improved methods and systems for scanning an intraoral cavityof a patient, and/or for automating the identification and analysis ofdental caries.

SUMMARY OF THE DISCLOSURE

Described herein are methods and apparatuses (e.g., devices and systems)that apply scans of both external and/or internal structures of teeth.These methods and apparatuses may generate and/or manipulate a model ofa subject's oral cavity (e.g. teeth, jaw, palate, gingiva, etc.) thatmay include both surface topography and internal features (e.g., dentin,dental filling materials (including bases and linings), cracks and/orcaries). Apparatuses for performing both surface and penetrativescanning of the teeth may include intraoral scanners for scanning intoor around a subject's oral cavity and that are equipped with a lightsource or light sources that can illuminate in two or more spectralranges: a surface-feature illuminating spectral range (e.g., visiblelight) and a penetrative spectral range (e.g. IR range, and particularly“near-IR,” including but not limited to 850 nm). The scanning apparatusmay also include one or more sensors for detecting the emitted light andone or more processors for controlling operation of the scanning and foranalyzing the received light from both the first spectral range and thesecond spectral range to generate a model of the subject's teethincluding the surface of the teeth and features within the teeth,including within the enamel (and/or enamel-like restorations) anddentin. The generated mode may be a 3D volumetric model or a panoramicimage.

As used herein, a volumetric model may include a virtual representationof an object in three dimensions in which internal regions (structures,etc.) are arranged within the volume in three physical dimensions inproportion and relative relation to the other internal and surfacefeatures of the object which is being modeled. For example, a volumetricrepresentation of a tooth may include the outer surface as well asinternal structures within the tooth (beneath the tooth surface)proportionately arranged relative to the tooth, so that a sectionthrough the volumetric model would substantially correspond to a sectionthrough the tooth, showing position and size of internal structures; avolumetric model may be section from any (e.g., arbitrary) direction andcorrespond to equivalent sections through the object being modeled. Avolumetric model may be electronic or physical. A physical volumetricmodel may be formed, e.g., by 3D printing, or the like. The volumetricmodels described herein may extend into the volume completely (e.g.,through the entire volume, e.g., the volume of the teeth) or partially(e.g., into the volume being modeled for some minimum depth, e.g., 2 mm,3 mm, 4 mm, 5 mm, 6 mm, 7 mm, 8 mm, 9 mm, 10 mm, 12 mm, etc.).

The methods described herein typically include methods for generating amodel of a subject's teeth typically generating a 3D model or renderingof the teeth that include both surface and internal features.Non-ionizing methods of imaging and/or detecting internal structures maybe used, such as taking images using a penetrating wavelength to viewstructures within the teeth by illuminating them using one or morepenetrative spectral ranges (wavelengths), including usingtrans-illumination (e.g., illuminating from one side and capturing lightfrom the opposite side after passing through the object), and/orsmall-angle penetration imaging (e.g., reflective imaging, capturinglight that has been reflected/scattered from internal structures whenilluminating with a penetrating wavelength). In particular, multiplepenetration images may be taken from the same relative position.Although traditional penetration imaging techniques (e.g.,trans-illumination) may be used, in which the angle between the lightemitter illumination direction and the detector (e.g., camera) viewangle is 90 degrees or 180 degrees, also described herein are methodsand apparatuses in which the angle is much smaller (e.g., between 0degrees and 25 degrees, between 0 degrees and 20 degrees, between 0degrees and 15 degrees, between 0 degrees and 10 degrees, etc.). Smallerangles (e.g., 0-15°) may be particularly beneficial because theillumination (light source) and sensing (detector(s), e.g., camera(s),etc.) may be closer to each other, and may provide a scanning wand forthe intraoral scanner that can be more easily positioned and movedaround a subject's teeth. These small-angle penetration images andimaging techniques may also be referred to herein as reflectiveillumination and/or imaging, or as reflective/scattering imaging. Ingeneral penetrating imaging may refer to any appropriate type ofpenetrating imaging unless otherwise specified, includingtrans-illumination, small-angle penetration imaging, etc. However, smallangles may also result in direct reflection from the surface of theobject (e.g., teeth), which may obscure internal structures.

The methods and apparatuses described here are particularly effective incombining a 3D surface model of the tooth or teeth with the imagedinternal features such as lesions (caries, cracks, etc.) that may bedetected by the use of penetration imaging by using an intraoral scannerthat is adapted for separate but concurrent (or nearly-concurrent)detection of both the surface and internal features. Combining surfacescanning and the penetration imaging may be performed by alternating orswitching between these different modalities in a manner that allows theuse of the same coordinate system for the two. Alternatively, bothsurface and penetrative scanning may be simultaneously viewed, forexample, by selectively filtering the wavelengths imaged to separate theIR (near-IR) light from the visible light. The 3D surface data maytherefore provide important reference and angle information for theinternal structures, and may allow the interpretation and analysis ofthe penetrating images that may otherwise be difficult or impossible tointerpret.

For example, described herein are methods for generating a model of asubject's teeth including the steps of: capturing three-dimensional (3D)surface model data of at least a portion of a subject's tooth using anintraoral scanner; taking a plurality of images into the tooth using apenetrative wavelength with the intraoral scanner; and forming a 3Dmodel of the tooth including internal structure using the 3D surfacemodel data and the plurality of images.

A method for generating a model of a subject's teeth may include:capturing three-dimensional (3D) surface model data of at least aportion of a subject's tooth with an intraoral scanner operating in afirst imaging modality, wherein the 3D surface model data has a firstcoordinate system; taking a plurality of images into the tooth with theintraoral scanner operating in a second imaging modality using apenetrative wavelength, wherein the plurality of images reference thefirst coordinate system; and forming a 3D model of the tooth includinginternal structures using the 3D surface model data and the plurality ofimages. In general, the capturing the first wavelength does notnecessarily capture images, but may directly capture a 3D surface scan.The second penetrating modalities may be captured as images processed asdescribed herein.

In general, capturing the 3D surface model data may include determininga 3D surface topology using any appropriate method. For example,determining a 3D surface topology may include using confocal focusing.Capturing the 3D surface model data may comprise using on or more of:confocal scanning, stereo vision or structured light triangulation.

Any of the methods and apparatuses described herein may be used tomodel, image and/or render a 3D image of a single tooth or region of atooth, multiple teeth, teeth and gums, or other intraoral structures,particularly from within a subject's mouth.

In general, the methods and apparatuses for performing them describedherein include 3D color intraoral scanning/scanners. For example, themethods may include capturing color intraoral 3D data.

As will be described in greater detail below, the method and apparatusesmay control the switching between collecting surface data and collectingpenetration imaging (penetrative) data. For example, any of thesemethods may include taking images using the penetrative wavelength asthe 3D surface model data is being captured, e.g., by switching betweenthe first imaging modality and the second (penetrative) imagingmodality.

The same sensor or a different sensor may be used to collect the surfaceand internal feature data. For example, taking the plurality of imagesmay comprise using a same sensor on the intraoral scanner to capture 3Dsurface model data and the plurality of images using the penetrativewavelength. Alternatively, a separate sensor or sensors may be used. Forexample, taking the plurality of images may comprise using a differentsensor on the intraoral scanner to capture 3D surface model data and theplurality of images using the penetrative wavelength.

As mentioned, taking images of the tooth using the penetrativewavelength (or penetrative spectral range) may include takingpenetration images at any angle between the illumination source and thesensor (e.g., detector or camera). In particular, internal feature(e.g., reflective imaging) data may be imaged using a small angleconfiguration, in which one or preferably more penetration images aretaken at different orientations relative to the tooth/teeth. Forexample, taking the plurality of images may comprise illuminating thetooth at an angle of between 0° and 15° relative to a sensor (e.g.,detector, camera, etc.) receiving the illumination from the tooth,reflecting off of the internal composition of the tooth/teeth. Takingthe plurality of images (e.g., penetration images such as thesesmall-angle penetration images) generally includes taking one or more(e.g., a plurality, including two or more, three or more, etc.)penetration images at different angles of the intraoral scanner relativeto the tooth over the same region of the tooth. Thus, the same internalregion of the tooth will appear in multiple different scans fromdifferent angles.

In general, any number of sensors may be included on the intraoralscanner, e.g., the wand of the intraoral scanner. Any appropriate sensorfor detecting and recording the appropriate spectral range(s) (e.g., oflight) may be used. Sensors may be referred to and may includedetectors, cameras, and the like. For example, taking a plurality ofimages may comprise using a plurality of sensors on the intraoralscanner to capture the plurality of images using the penetrativewavelength.

The illumination used to take a penetration image is generallypenetrative, so that it may at least partially penetrate and passthrough the enamel and dentin of the teeth. Penetrative wavelengths oflight may include generally infrared (and particularly near infrared)light. For example, light in the range of 700 to 1090 nm (e.g., 850 nm)may be used. Other wavelengths and ranges of wavelengths may be used,including wavelengths shorter than the visible spectrum. Thus, takingthe plurality of images may comprise illuminating the tooth withinfrared light. Taking the plurality of images (e.g., penetrationimages) may include illuminating the tooth with one or more of whitelight (including but not limited to white light trans-illumination),UV/Blue fluorescence and red light fluorescence.

The illumination used to take a penetration image can be consideredsemi-penetrative in the sense that internal tooth regions (e.g., pointsor voxels) may be visible from only a few camera positions andorientations; the point may be obstructed by other structures in someimages which include the volume point in their field of view. In thatsense, images that include the volume point in their field of view maynot image this volume point. Thus, the methods and apparatuses describedherein may take into account the high masking of volume points, unlikeother penetrative scanning techniques such as CT, which uses X-rayimaging in which no masking occurs.

In general, any appropriate technique may be used to form the 3D modelsof the tooth including the (combined) surface and internal structuresfrom the penetration imaging. These 3D models may be referred to ascombined 3D surface/volume models, 3D volumetric surface models, orsimply “3D models,” or the like. As mentioned, both the surface data andthe penetration imaging data may generally be in the same coordinatesystem. The two may be combined by using the common coordinate system.In some variations the surface data may be expressed as a surface modeland the internal features added to this model. In some variations thedata may be reconstructed into a three-dimensional model concurrently(after adding together). One or both datasets may be separately modified(e.g., filtered, subtracted, etc.). For example, forming the 3D model ofthe tooth including internal structures may comprise combing the 3Dsurface model data with an internal structure data (including volumetricdata). Forming the 3D model of the tooth including internal structuresmay comprise combining the plurality of penetration images, wherein theplurality of penetration images may be taken from different angles usingthe intraoral scanner.

In any of the methods and apparatuses configured to perform thesemethods described herein, the data may be analyzed automatically ormanually by the system. In particular, the method and apparatusesdescribed herein may include examining internal features and/oridentifying features of interest, including crack and caries. Featuresmay be recognized based on feature-recognition criterion (e.g., dark orlight regions in the penetration images), pattern-recognition, machinelearning, or the like. Features may be marked, including coloring,labeling or the like. Feature may be marked directly in the 3D model, onthe penetration image, or in a data structure that references (e.g.,shares a coordinate system with) the 3D model of the tooth formed by themethods and apparatuses described herein.

Also described herein are apparatuses configured to perform any of themethods described. For example, described herein are intraoral scanningsystems for generating a model of a subject's teeth that include: ahand-held wand having at least one sensor and a plurality of lightsources, wherein the light sources are configured to emit light at afirst spectral range and a second spectral range, wherein the secondspectral range is penetrative; and one or more processors operablyconnected to the hand-held wand, the one or more processors configuredto: generate a three-dimensional (3D) surface model of at least aportion of a subject's tooth using light from a first spectral range;and generate a 3D model of the subject's tooth including internalstructures based on the 3D surface model and on a plurality of imagestaken at the second spectral range showing internal structures.

An intraoral scanning system for generating a model of a subject's teethmay include: a hand-held wand having at least one sensor and a pluralityof light sources, wherein the light sources are configured to emit lightat a first spectral range and a second spectral range, further whereinthe second spectral range is penetrative; and one or more processorsoperably connected to the hand-held wand, the one or more processorsconfigured to: determine surface information by using light in the firstspectral range sensed by the hand-held wand, using a first coordinatesystem; generate a three-dimensional (3D) surface model of at least aportion of a subject's tooth using the surface information; take aplurality of images in the second spectral range, wherein the imagesreference the first coordinate system; and generate a 3D model of thesubject's tooth including internal structures based on the 3D surfacemodel and the a plurality of images.

Also described herein are methods of generating a model of a subject'steeth that include both surface and internal structures in which thesame intraoral scanner is cycled between different modalities such asbetween surface scanning and penetration; additional modalities (e.g.,laser florescence, etc.) may also alternatively be included. In general,although the examples described herein focus on the combination ofsurface and penetration, other internal scanning techniques (e.g., laserflorescence) may be used instead or in addition to the internal featureimaging described herein.

For example, described herein are methods of generating a model of asubject's teeth including both surface and internal structures includingthe steps of: using a hand-held intraoral scanner to scan a portion of asubject's tooth using a first modality to capture three-dimensional (3D)surface model data of the tooth; using the hand-held intraoral scannerto scan the portion of the subject's tooth using a second modality toimage into the tooth using a penetrative wavelength to capture internaldata of the tooth; cycling between the first modality and the secondmodality, wherein cycling rapidly switches between the first modalityand the second modality so that images using the penetrative wavelengthshare a coordinate system with the 3D surface model data captured in thefirst modality.

Any of the methods described herein may include automatically adjustingthe duration of time spent scanning in first modality, the duration oftime spent in the second modality, or the duration of time spent in thefirst and the second modality when cycling between the first modalityand the second modality. For example, any of these methods may includeautomatically adjusting a duration of time spent scanning in firstmodality, the duration of time spent in the second modality, or theduration of time spent in the first and the second modality when cyclingbetween the first modality and the second modality based on the captured3D surface model data, the internal data, or both the 3D surface modeldata and the internal data. Thus, a method of generating a model of asubject's teeth may include: using a hand-held intraoral scanner to scana portion of a subject's tooth using a first modality to capturethree-dimensional (3D) surface model data of the tooth; using thehand-held intraoral scanner to scan the portion of the subject's toothusing a second modality to image into the tooth using a penetrativewavelength to capture internal data of the tooth; cycling between thefirst modality and the second modality using a scanning scheme whereincycling rapidly switches between the first modality and the secondmodality so that the internal data uses the same coordinate system asthe 3D surface model data captured in the first modality; and adjustingthe scanning scheme based on the captured 3D surface model data, theinternal data, or both the 3D surface model data and the internal data.

The scanning scheme adjustment may comprise adjusting based ondetermination of the quality of the captured 3D surface model data.Adjusting the scanning scheme may comprise automatically adjusting thescanning scheme, and/or adjusting a duration of scanning in the firstmodality and/or adjusting a duration of scanning in the second modality.

Any of these methods may include combining the 3D surface model data andthe internal data of the tooth to form a 3D model of the tooth.

As mentioned above, capturing the 3D surface model data may includedetermining a 3D surface topology using confocal focusing/confocalscanning, stereo vision or structured light triangulation.

In general, cycling may comprise cycling between the first modality, thesecond modality, and a third modality, wherein cycling rapidly switchesbetween the first modality, the second modality and the third modalityso that images using the penetrative wavelength share a coordinatesystem with the 3D surface model captured in the first modality. Thethird modality may be another penetrative modality or a non-penetrativemodality (e.g., color, a visual image the subject's tooth, etc.).

Using the hand-held intraoral scanner to scan the portion of thesubject's tooth using the second modality may include illuminating thetooth at an angle of between 0° and 15° relative to a direction of viewof the sensor receiving the illumination (e.g., small angleillumination). The step of using the hand-held intraoral scanner to scanthe portion of the subject's tooth using the second modality may includetaking a plurality of penetration images at a plurality of differentangles between an illumination source and a sensor and/or at a pluralityof different positions or angles relative to the tooth so that the sameinternal region of the tooth is imaged from different angles relative tothe tooth.

As mentioned, any appropriate penetrative wavelength may be used,including infrared (e.g., near infrared). For example using thehand-held intraoral scanner to scan the portion of the subject's toothusing the second modality may comprise illuminating with one or more of:white light trans-illumination, UV/Blue fluorescence, and red lightfluorescence.

Also described herein are intraoral scanning systems for generating amodel of a subject's teeth that are configured to cycle between scanningmodes. For example, described herein are intraoral scanning systemscomprising: a hand-held intraoral wand having at least one sensor and aplurality of light sources, wherein the light sources are configured toemit light at a first spectral range and at a second spectral range,further wherein the second spectral range is penetrative; and one ormore processors operably connected to the hand-held intraoral wand, theone or more processors configured to cause the wand to cycle between afirst mode and a second mode, wherein in the first mode the wand emitslight at the first spectral range for a first duration and the one ormore processors receives three dimensional (3D) surface data inresponse, and wherein in the second mode the wand emits light at thesecond spectral range for a second duration and the one or moreprocessors receives image data in response.

An intraoral scanning system for generating a model of a subject's teethmay include: a hand-held intraoral wand having at least one sensor and aplurality of light sources, wherein the light sources are configured toemit light at a first spectral range and at a second spectral range,further wherein the second spectral range is penetrative; and one ormore processors operably connected to the wand, the one or moreprocessors configured to cause the wand to cycle between a first modeand a second mode, wherein in the first mode the wand emits light at thefirst spectral range for a first duration and the one or more processorsreceives three dimensional (3D) surface data in response, and wherein inthe second mode the wand emits light at the second spectral range for asecond duration and the one or more processors receives image data inresponse; wherein the one or more processors is configured to adjustingthe first duration and the second duration based on the received 3Dsurface data, the received image data, or both the 3D surface data andthe image data. In any of the apparatuses described herein, one mode maybe the surface scanning (3D surface), which may be, for example, at 680nm. Another mode may be a penetrative scan, using, e.g., near-IR light(e.g., 850 nm). Another mode may be color imaging, using white light(e.g., approximately 400 to 600 nm).

Penetration imaging methods for visualizing internal structures using ahand-held intraoral scanner are also described. Thus, any of the generalmethods and apparatuses described herein may be configured specificallyfor using penetration imaging data to model a tooth or teeth to detectinternal features such as crack and caries. For example, a method ofimaging through a tooth to detect cracks and caries may include: takinga plurality of penetration images through the tooth at differentorientations using a hand-held intraoral scanner in a first position,wherein the intraoral scanner is emitting light at a penetrativewavelength; determining surface location information using the intraoralscanner at the first position; and generating a three-dimensional (3D)model of the tooth using the plurality of penetration images and thesurface location information.

Generating a 3D model of the tooth may comprise repeating the steps oftaking the plurality of penetration images and generating the 3D modelfor a plurality of different locations.

Taking the plurality of penetration images through the tooth atdifferent orientations may include taking penetration images in whicheach penetration image is taken using either or both of: a differentillumination source or combination of illumination sources on theintraoral scanner emitting the penetrative wavelength or a differentimage sensor on the intraoral scanner taking the image.

In some variations taking the plurality of penetration images maycomprise taking three or more penetration images.

Taking the plurality of penetration images through the tooth surface atdifferent orientations may comprises taking penetration images usingsmall angle illumination/viewing, for example, wherein, for eachpenetration image, an angle between emitted light and light received byan image sensor is between 0 and 15 degrees. For example, a method ofimaging through a tooth to detect cracks and caries may include:scanning a tooth from multiple positions, wherein scanning comprisesrepeating, for each position: taking a plurality of penetration imagesthrough the tooth at different orientations using an intraoral scanner,wherein the intraoral scanner is emitting light at a penetrativewavelength and wherein, for each penetration image, an angle betweenemitted light and light received by an image sensor is between 0 and 15degrees, and determining surface location information using theintraoral scanner; and generating a three-dimensional (3D) model of thetooth using the penetration images and the surface location information.

As mentioned above, in addition to the apparatuses (e.g., scanningapparatuses, tooth modeling apparatuses, etc.) and methods of scanning,modeling and operating a scanning and/or modeling apparatus, alsodescribed herein are methods of reconstructing volumetric structuresusing images generated from one or more penetrative wavelengths.

For example, described herein are methods of reconstructing a volumetricstructure from an object including semi-transparent strongly scatteringregions (e.g., a tooth) for a range of radiation wavelengths. The methodmay include illuminating the object with a light source that is emitting(e.g., exclusively or primarily radiating) a penetrating wavelength,taking a plurality of images of the object with a camera sensitive tothe penetrating wavelength (e.g., recording in the range of radiationwavelengths), receiving location data representing a location of thecamera relative to the object for each of the plurality of images,generating for each point in a volume an upper bound on a scatteringcoefficient from the plurality of images and the location data, andgenerating an image of the object from the upper bound of scatteringcoefficients for each point. The penetrating wavelength of light appliedto the object may be emitted from substantially the same direction asthe camera. The image or images generated may illustrate features withinthe volume of the object, and the image may also include (or be modifiedto include) the outer boundary of the object, as well as the internalstructure(s).

As used herein, a tooth may be described as an object includingsemi-transparent strongly scattering region or regions; in general,teeth may also include strong scattering regions (such as dentine), andlightly scattering, highly transparent regions (such as the enamel) atnear-IR wavelengths. Teeth may also include regions having intermedia ormixed scattering properties, such as caries. The methods and apparatusesfor performing volumetric scans described herein are well suited formapping these different regions in the tooth/teeth.

A method of reconstructing a volumetric structure from an objectincluding semi-transparent strongly scattering regions for a range ofradiation wavelengths may include: taking a plurality of images of theobject with a camera in the range of radiation wavelengths, whereinlighting for the plurality of images is projected substantially from adirection of the camera, receiving location data representing a locationof the camera relative to the object for each of the plurality ofimages, generating for each point in a volume an upper bound on ascattering coefficient from the plurality of images and the locationdata, and generating an image of the object from the upper bound ofscattering coefficients for each point.

The range of radiation wavelengths may be infrared or near infraredwavelength(s).

Any of these methods may also include receiving surface datarepresenting an exterior surface of the object, wherein the generatingstep is performed for each point in the volume within the exteriorsurface of the object.

The object may comprise a tooth, having an exterior enamel surface andan interior dentin surface. Teeth are just one type of object includingsemi-transparent strongly scattering regions; other examples may includeother both tissues (including soft and/or hard tissues), e.g., bone,etc. These objects including semi-transparent strongly scatteringregions may include regions that are typically semi-transparent andstrongly scattering for the penetrative wavelengths (e.g., the infraredor near infrared wavelengths), as described herein.

The location data may generally include position and orientation data ofthe camera at the time of capturing each of the plurality of images. Forexample, the location data may comprise three numerical coordinates in athree-dimensional space, and pitch, yaw, and roll of the camera.

Generating for each point in the volume the upper bound on scatteringcoefficients may comprise projecting each point of a 3D grid of pointscorresponding to the volume of the object onto each of the pluralityimages using a first calibration, producing a list of intensity valuesfor each projected point, converting each intensity value on the list ofintensity values to a scattering coefficient according to a volumeresponse, and storing a minimum scattering coefficient value for eachgrid point from the list of scattering coefficient values.

For example, the first calibration may comprise a fixed pattern noisecalibration to calibrate for sensor issues and image ghosts of thecamera. The first calibration may comprise a camera calibration thatdetermines a transformation for the camera that projects known points inspace to points on an image.

Also described herein are methods of reconstructing a volumetricstructure from a tooth, semi-transparent in a range of radiationwavelengths, the method comprising receiving, in a processor, arepresentation of a surface of the tooth in a first coordinate system,receiving, in the processor, a plurality of images of the tooth in therange of radiation wavelengths, the plurality of images taken withlighting projected substantially from a direction of a camera,receiving, in the processor, location data representing a location ofthe camera for each of the plurality of images, projecting each point ofa grid of points corresponding to a volume within the surface of thetooth onto each of the plurality images using a first calibration,producing a list of intensity values for each projected point,converting each intensity value on the list of intensity values to ascattering coefficient according to a volume response, and storing aminimum scattering coefficient for each point into a list of minimumscattering coefficients.

Any of these methods may further comprise producing an image from thelist of minimum scattering coefficients.

The location data may comprise position and orientation data of thecamera (or cameras) at the time of capturing each of the plurality ofimages.

The first calibration may comprise a fixed pattern noise calibration tocalibrate for sensor issues and image ghosts of the camera. In someembodiments, the first calibration may comprise a camera calibrationthat determines a transformation for the camera that projects knownpoints in space to points on an image.

The method may further comprise receiving surface data representing anexterior surface of the object, wherein the projecting step is performedfor each point inside the volume within the exterior surface of theobject.

The grid of points may comprise a cubic grid.

Any of the methods described herein may be embodied as software,firmware and/or hardware. For example, any of these methods may beconfigured as non-transitory computing device readable medium havinginstructions stored thereon for performing the method.

For example, a non-transitory computing device readable medium havinginstructions stored thereon for reconstructing a volumetric structurefrom a tooth that is semi-transparent in a range of radiationwavelengths is described. The instructions may be executable by aprocessor to cause a computing device to receive a representation of asurface of the tooth in a first coordinate system, receive a pluralityof images of the tooth in the range of radiation wavelengths, theplurality of images taken with lighting projected substantially from adirection of a camera, receive location data representing a location ofthe camera for each of the plurality of images, project each point of agrid of points corresponding to a volume of the tooth onto each of theplurality of images using a first calibration, produce a list ofintensity values for each projected point, convert each intensity valueon the list of intensity values to a scattering coefficient according toa volume response, and store a minimum scattering coefficient for eachpoint into a list of minimum scattering coefficients, and produce animage from the list of minimum scattering coefficients.

The location data may comprise position and orientation data of thecamera at the time of capturing each of the plurality of near-infraredimages. The location data may comprise three numerical coordinates in athree-dimensional space, and pitch, yaw, and roll of the camera.

The first calibration may comprise a fixed pattern noise calibration tocalibrate for sensor issues and image ghosts of the camera. The firstcalibration may comprise a camera calibration that determines atransformation for the camera that projects known points in space topoints on an image.

The grid of points may be inside the tooth; as mentioned, the grid ofpoints may comprise a cubic grid.

Alternatively or additionally to the use of scattering coefficients, anyappropriate method of forming the internal structures of the patient'steeth using the penetrative wavelength images. For example, any of theapparatuses (e.g., systems, devices, software, etc.) and methodsdescribed herein may use the two-dimensional penetrative images alongwith position and/or orientation information about the scanner relativeto the object being imaged (e.g., the teeth) to segment the 2Dpenetrative images to form a three-dimensional model of the teethincluding an internal structure from within the teeth. As described, apenetrative image may refer to an images taken with a near-IR and/or IRwavelength), penetrating into the object. The position and/ororientation of the scanner may be a proxy for the position and/ororientation of the camera taking the images which is one the scanner(e.g., on a handheld wand).

For example, described herein are methods of modeling a subject's teeth,comprising: capturing, with an intraoral scanner, a plurality of imagesof an interior of the subject's teeth and a position and orientation ofthe intraoral scanner specific to each image of the plurality of images;segmenting the plurality of images to form an internal structurecorresponding to a structure within the subject's teeth; using theposition and orientation of the plurality of images to project theinternal structure onto a three-dimensional model of the subject'steeth; and displaying the three-dimensional model of the subject's teethincluding the internal structure.

In any of these methods and apparatuses, the 3D surface model may beconcurrently captured using a non-penetrative wavelength (e.g., surfacescan) while capturing the penetrative images. For example, capturing maycomprise capturing surface images of the subject's teeth while capturingthe plurality of images of the interior of the subject's teeth. Themethod may also include forming the three dimensional model of thesubject's teeth from the captured surface images. For example, formingthe three dimensional model of the subject's teeth may comprisedetermining a three-dimensional surface topology using confocalfocusing. Capturing the surface images of the subject's teeth maycomprise using confocal scanning, stereo vision or structured lighttriangulation.

In general, the same device (e.g., scanner) may model and/or display the3D representation of the teeth, including the internal structures,alternatively or additionally a separate processor (e.g., remote to thescanner) may be used. Any of these methods may also include storingand/or transmitting plurality of penetrative images and the position andorientation of the intraoral scanner while capturing the plurality oftwo-dimensional images, including transmitting to a remote processor forperforming the segmentation and later steps.

In any of the methods and apparatuses described herein, the 3D modelincluding the internal structure(s) may be displayed while the scanneris operating. This may advantageously allow the user to see, inreal-time or near real-time the internal structure(s) in the subject'steeth. Thus, any of these methods may include displaying thethree-dimensional model as the images are captured.

Segmenting the plurality of images may comprise applying edge detectionto the plurality of images to identify closed boundaries within theplurality of images. Segmenting the plurality of images may compriseforming a volumetric density map from the plurality of images toidentify the internal structure. Segmenting the volumetric density mapmay include segmenting by identifying one or more iso-surfaces withinthe volumetric density map to identify the internal features. Any ofthese methods may include segmenting the volumetric density map toidentify the internal feature (e.g., cracks, caries, dental fillings,dentin, etc.).

For example, an intraoral scanning apparatus configured to generate amodel of a subject's teeth may include: an intraoral scanner having aplurality of light sources and a position and orientation sensor,wherein the light sources are configured to emit light at a firstspectral range and at a second spectral range, further wherein thesecond spectral range is penetrative; and a processor operably connectedto the intraoral scanner, the one or more processors configured to causethe scanner to capture a plurality of images and position andorientation of the intraoral scanner corresponding to each of theplurality of images when the intraoral scanner is emitting light at thesecond spectral range; wherein the processor is further configured tosegment the plurality of images to form an internal structurescorresponding to a structure within the subject's teeth, and to displayor transmit a three-dimensional model of the subject's teeth includingthe internal structure.

The processors may be configured to segment the plurality of images byapplying edge detection to the plurality of images to identify closedboundaries within the plurality of images. The processor may beconfigured to segment the plurality of images by forming a pixel densitymap from the plurality of images to identify the internal structure. Theprocessor may be configured to identify closed segments within the pixeldensity map to identify the internal structure.

Also described herein are non-transitory computing device readablemedium having instructions stored thereon that are executable by aprocessor to cause an intraoral scanning apparatus to: capture aplurality of images using a penetrative wavelength of light and aposition and orientation of the intraoral scanner specific to each imageof the plurality of images; segment the plurality of images to form aninternal structure corresponding to a structure within a subject'steeth; use the position and orientation of the intraoral scannerspecific to each image to project the internal structure onto athree-dimensional model of the subject's teeth; and display thethree-dimensional model of the subject's teeth including the internalstructure.

The non-transitory computing device readable medium having instructionsmay be further configured to cause the intraoral scanning apparatus tosegment the plurality of images by applying edge detection to theplurality of images to identify closed boundaries within the pluralityof images. The non-transitory computing device readable medium havinginstructions may be further configured to cause the intraoral scanningapparatus to segment the plurality of images by forming a pixel densitymap from the plurality of images to form the internal structure. Thenon-transitory computing device readable medium having instructions maybe further configured to cause the intraoral scanning apparatus tosegment the plurality of images by identifying closed segments withinthe pixel density map to form the internal structure.

Also described herein are non-transitory computing device readablemedium having instructions stored thereon that are executable by aprocessor to cause a computing device to: receive, from a scanner,three-dimensional surface model data of a subject's teeth; receive, fromthe scanner, a plurality of images of an interior of the subject's teethand position and orientation of the intraoral scanner specific to eachimage of the plurality of images; segment the plurality of images toform an internal structure of the subject's teeth; project the internalstructure of the subject's teeth onto the three-dimensional surfacemodel; and display the three-dimensional surface model showing theinternal structure.

For example, described herein are methods for generating athree-dimensional (3D) volumetric model of a subject's teeth using anintraoral scanner, the method comprising: capturing 3D surface modeldata of at least a portion of the subject's teeth using an intraoralscanner as the intraoral scanner is moved over the teeth; taking aplurality of images into the teeth using a near-infrared (near-IR)wavelength with the intraoral scanner as the intraoral scanner is movedover the teeth so that multiple images of a same internal region of theteeth are imaged; determining, for each of the plurality of images intothe teeth, a position of the intraoral scanner relative to the subject'steeth using the 3D surface model data; and forming the 3D volumetricmodel of the subject's teeth including internal features using theplurality of images and the position of the intraoral scanner relativeto the subject's teeth.

A method for generating a three-dimensional (3D) volumetric model of asubject's teeth using an intraoral scanner may include: capturing 3Dsurface model data of at least a portion of the subject's teeth using anintraoral scanner as the intraoral scanner is moved over the teeth;taking a plurality of images into the teeth using a near-infrared(near-IR) wavelength as the intraoral scanner is moved over the teeth byemitting a near-IR light from the intraoral scanner in a firstpolarization, and detecting, in an image sensor in the intraoralscanner, the near-IR light returning to the intraoral scanner, whereinthe near-IR light returning to the intraoral scanner is filtered toremove specular reflection by filtering near-IR light in the firstpolarization from the near-IR light returning to the intraoral scannerbefore it reaches the image sensor; determining, for each of theplurality of images into the teeth, a position of the intraoral scannerrelative to the subject's teeth when each of the plurality of images iscaptured, using the 3D surface model data; and forming the 3D volumetricmodel of the subject's teeth including internal features using theplurality of images and the position of the intraoral scanner relativeto the subject's teeth.

In any of these methods and apparatuses, the near-IR light returning tothe intraoral scanner may be filtered to remove specular reflection byfiltering all or nearly all of the near-IR light in the firstpolarization from the near-IR light returning to the intraoral scannerbefore it reaches the image sensor.

Also described herein are intraoral scanners scan both surface andinternal structures. For example, an intraoral scanning system forgenerating a three-dimensional (3D) volumetric model of a subject'steeth may include: a hand-held wand having at least one image sensor anda plurality of light sources, wherein the light sources are configuredto emit light at a first spectral range and a second spectral range,wherein the second spectral range is within near-infrared (near-IR)range of wavelengths; and one or more processors operably connected tothe hand-held wand, the one or more processors configured to: capture 3Dsurface model data of at least a portion of the subject's teeth as theintraoral scanner is moved over the teeth; take a plurality of imagesinto the teeth using light in the second spectral range as the intraoralscanner is moved over the teeth so that multiple images of a sameinternal region of the teeth are imaged; determine, for each of theplurality of images into the teeth, a position of the hand-held wandrelative to the subject's teeth using the 3D surface model data; andform the 3D volumetric model of the subject's teeth including internalfeatures using the plurality of images and the position of the intraoralscanner relative to the subject's teeth.

An intraoral scanning system for generating a three-dimensional (3D)volumetric model of a subject's teeth may include: a hand-held wandhaving at least one image sensor and a plurality of light sources,wherein the light sources are configured to emit light at a firstspectral range and a second spectral range, wherein the second spectralrange is within near-infrared (near-IR) range of wavelengths; a filterin front of the image sensor configured to filter light in the secondspectral range and the first polarization; and one or more processorsoperably connected to the hand-held wand, the one or more processorsconfigured to: capture 3D surface model data of at least a portion ofthe subject's teeth as the intraoral scanner is moved over the teeth;take a plurality of images into the teeth using light in the secondspectral as the intraoral scanner is moved over the teeth by emitting anear-IR light from the intraoral scanner in a first polarization, anddetecting, in an image sensor in the intraoral scanner, the near-IRlight returning to the intraoral scanner, wherein the near-IR lightreturning to the intraoral scanner is filtered to remove specularreflection by filtering near-IR light in the first polarization from thenear-IR light returning to the intraoral scanner before it reaches theimage sensor; determine, for each of the plurality of images into theteeth, a position of the hand-held wand relative to the subject's teethusing the 3D surface model data; and form the 3D volumetric model of thesubject's teeth including internal features using the plurality ofimages and the position of the intraoral scanner relative to thesubject's teeth.

Also described herein are methods of imaging cracks and caries in teeth.For example, described herein are methods of imaging into a subject'steeth to detect cracks and caries using an intraoral scanner, the methodcomprising: scanning the intraoral scanner over the subject's teeth;taking a plurality of near-infrared (near-IR) images into the subject'steeth at different orientations using the intraoral scanner emittingboth a near-IR wavelength and a non-penetrative wavelength; determininga position of the intraoral scanner relative to the subject's teeth foreach location of an image from the plurality of near-IR images using thenon-penetrative wavelength; and generating a three-dimensional (3D)volumetric model of the subject's teeth using the plurality of near-IRimages and the position of the intraoral scanner relative to thesubject's teeth for each near-IR image of the plurality of near-IRimages.

Any of these methods may include analyzing the volumetric model toidentify a crack or caries (or other internal regions of the teeth).

For example, a method of imaging through a subject's teeth to detectcracks and caries may include: scanning the subject's teeth frommultiple positions, wherein scanning comprises repeating, for eachposition: taking a plurality of near-infrared (near-IR) images into theteeth at different orientations using an intraoral scanner, wherein theintraoral scanner is emitting light at a near-IR wavelength in a firstpolarization and wherein, for each near-IR image, an angle betweenemitted light and light received by an image sensor is between 0 and 15degrees, further wherein received near-IR light is filtered to blocknear-IR light in the first polarization, and determining a position ofthe intraoral scanner relative to the subject's teeth for each locationof an image from the plurality of near-IR images using; and generating athree-dimensional (3D) volumetric model of the tooth using thepenetration images and the surface location information.

Also described herein are methods of using scattering coefficients togenerate internal images of tooth based on penetrating images and camerasensor location. For example, a method of forming a three-dimensional(3D) volumetric model of a subject's teeth may include: taking aplurality of near-infrared (near-IR) images of the subject's teeth witha camera sensor, wherein the near-IR lighting for the plurality ofnear-IR images is projected substantially from a direction of the camerasensor; receiving location data representing a location of the camerarelative to the subject's teeth for each of the plurality of near-IRimages; generating, for each point in a volume, an upper bound on ascattering coefficient from the plurality of near-IR images and thelocation data; combining the upper bound of scattering coefficients foreach point in a volume to form a 3D volumetric model of the subject'steeth; and outputting the 3D volumetric model of the subject's teeth.

Any of these methods may include forming an iso-surface from the 3Dvolumetric model of the subject's teeth. The iso-surface may be formedby selecting a threshold or range of values of the scatteringcoefficients. Sub-ranges may correspond to different internal regions(e.g., structures). For example, outputting may comprise forming aniso-surface corresponding to an interior dentin surface from the 3Dvolumetric model of the subject's teeth.

A method of reconstructing a volumetric structure from a tooth, whereinthe tooth is semi-transparent in a range of radiation wavelengths, mayinclude: receiving, in a processor, a representation of a surface of thetooth in a first coordinate system; receiving, in the processor, aplurality of images of the tooth taken by a camera in the range ofradiation wavelengths, the plurality of images taken with lightingprojected substantially from a direction of the camera; receiving, inthe processor, location data representing a location of the camera foreach of the plurality of images; projecting each point of a grid ofpoints corresponding to a volume within the surface of the tooth ontoeach of the plurality images using a first calibration; producing a listof intensity values for each projected point; converting each intensityvalue on the list of intensity values to a scattering coefficientaccording to a volume response; and storing a minimum scatteringcoefficient for each point into a list of minimum scatteringcoefficients.

Any of these methods may be embodied in an apparatus, includingsoftware, hardware and/or firmware for performing the method. Forexample, described herein are non-transitory computing device readablemedium having instructions stored thereon for reconstructing avolumetric structure from a tooth that is semi-transparent in a range ofradiation wavelengths, wherein the instructions are executable by aprocessor to cause a computing device to: receive a representation of asurface of the tooth in a first coordinate system; receive a pluralityof images of the tooth taken by a camera in the range of radiationwavelengths, the plurality of images taken with lighting projectedsubstantially from a direction of the camera; receive location datarepresenting a location of the camera for each of the plurality ofimages; project each point of a grid of points corresponding to a volumeof the tooth onto each of the plurality of images using a firstcalibration; produce a list of intensity values for each projectedpoint; convert each intensity value on the list of intensity values to ascattering coefficient according to a volume response; and store aminimum scattering coefficient for each point from the scatteringcoefficients; and output an image produced from the list of minimumscattering coefficients.

Also described herein are methods of forming the internal structuresusing segmentation. For example, a method of modeling a subject's teeth,may include: capturing, with an intraoral scanner, a plurality of imagesof an interior of the subject's teeth and a position and orientation ofthe intraoral scanner specific to each image of the plurality of images;segmenting the plurality of images to form an internal structurecorresponding to a structure within the subject's teeth; using theposition and orientation of the plurality of images to project theinternal structure onto a three-dimensional model of the subject'steeth; and displaying the three-dimensional model of the subject's teethincluding the internal structure.

Also described herein are intraoral scanning apparatus configured togenerate a model of a subject's teeth, the apparatus comprising: anintraoral scanner having a plurality of light sources and a position andorientation sensor, wherein the light sources are configured to emitlight at a first spectral range and at a second spectral range, furtherwherein the second spectral range is penetrative; and a processoroperably connected to the intraoral scanner, the one or more processorsconfigured to cause the scanner to capture a plurality of images andposition and orientation of the intraoral scanner corresponding to eachof the plurality of images when the intraoral scanner is emitting lightat the second spectral range; wherein the processor is furtherconfigured to segment the plurality of images to form an internalstructures corresponding to a structure within the subject's teeth, andto display or transmit a three-dimensional model of the subject's teethincluding the internal structure.

Also described herein are non-transitory computing device readablemedium having instructions stored thereon that are executable by aprocessor to cause an intraoral scanning apparatus to: capture aplurality of images using a penetrative wavelength of light and aposition and orientation of the intraoral scanner specific to each imageof the plurality of images; segment the plurality of images to form aninternal structure corresponding to a structure within a subject'steeth; use the position and orientation of the intraoral scannerspecific to each image to project the internal structure onto athree-dimensional model of the subject's teeth; and display thethree-dimensional model of the subject's teeth including the internalstructure.

Also described herein are methods for forming 3D volumes (includingvolumetric volumes) of teeth. For example, described herein are methodscomprising: receiving data associated with an intraoral scan of asubject; determining, from the received data, at least a portion of avolume of a first internal feature of a tooth of the subject;determining, from the received data, at least a portion of a volume of asecond internal feature of the tooth of the subject, the first internalfeature differing from the second internal feature; mapping the portionof the volume of the first internal feature with the portion of thevolume of the second internal feature; outputting a 3D volume of theportion of the volume of the first internal feature with the portion ofthe volume of the second internal feature.

The received data may comprise data from tooth surface penetratingintraoral scan of the subject. The received data may further comprisedata from a tooth surface intraoral scan of the subject.

The method may also include determining, from the received data, asurface of the tooth of the subject; mapping the surface of the toothwith the portion of the volume of the first internal feature and theportion of the volume of the second internal feature; and outputting the3D volume with the surface of the tooth with the portion of the volumeof the first internal feature and the portion of the volume of thesecond internal feature.

The received data may further comprise data from a tooth surface colorintraoral scan of the subject.

The method may also comprise, determining, from the received data, acolor of the surface of the tooth of the subject; mapping the color ofthe surface of the tooth to the surface of the tooth; and outputting the3D volume with the surface of the tooth and the color of the surface ofthe tooth.

The first internal feature of the tooth may comprise a dentin of thetooth and the second internal feature of the tooth comprises an enamelof the tooth. The intraoral scan may comprise a second intraoral scan ofthe subject; and wherein the method further comprises receiving dataassociated with a prior intraoral scan of the subject; determining fromthe received data associated with the prior intraoral scan of thesubject, at least a portion of a volume of the enamel or the dentin; anddetermining a volume change of the enamel or the dentin by comparing theportion of the volume of the enamel or the dentin determined from thereceived data associated with the second intraoral scan and the portionof the volume of the enamel or the dentin determined from the receiveddata associated with the prior intraoral scan; and outputting thedetermined volume change.

The method may also include detecting a dental caries of the tooth bycomparing the second internal feature and the first internal feature andoutputting a signal to the user associated with the detected dentalcaries. Comparing the second internal feature and the second internalfeature may comprise analyzing whether the volume of the second internalfeature extends from a surface of the volume of the first internalfeature. Analyzing may comprise determining whether the volume of thesecond internal feature extends from the surface of the volume of thefirst internal feature and to a portion of the second internal featureassociated with the dentin.

The method may also include calculating a volume of the second internalfeature that extends from the surface of the volume of the firstinternal feature and outputting a signal associated with the calculatedvolume.

Also described are method comprising: receiving data associated with anintraoral scan of a subject; determining, from the received data, avolume of a dental caries of a tooth of the subject; quantifying thevolume of the dental caries of the tooth of the subject; and outputtinga signal associated with the quantified volume of the dental caries ofthe tooth of the subject.

The method may also include determining, from the received data, avolume of an enamel of the tooth of the subject; mapping the volume ofthe enamel to the volume of the dental caries; and outputting a 3Dvolume of the mapped volumes of the enamel and the dental caries to auser. For example, determining, from the received data, a volume of adentin of the tooth of the subject; mapping the volume of the dentin tothe volume of the enamel and the volume of the dental caries; andoutputting the 3D volume of the mapped volumes of the enamel and thedental caries with the volume of the dentin.

The intraoral scan of the subject may comprise a second intraoral scanof the subject and wherein the method further comprises receiving dataassociated with a prior intraoral scan of the subject; determining, fromthe received data associated with the prior intraoral scan of thesubject, a prior volume of the dental caries of the tooth of thesubject; outputting a signal associated with a difference in volumebetween the volume of the dental caries and the prior volume of thedental caries. The method may also comprise outputting a 3D model of thevolume of the dental caries of the tooth of the subject.

Also described herein are trans-illumination adapter sleeve device foran intraoral scanner, the device comprising: a sleeve body configured tofit over a wand of an intraoral scanner, the sleeve body comprising alight-passing region at a distal end of the sleeve body configured toallow near-infrared (near-IR) light to pass through the sleeve; a firstwing region extending from the distal end of the sleeve body adjacent tothe light-passing region; and a near-IR light source configured to emitnear-IR light from the first wing region. The near-IR light source maybe configured to emit near-IR light transverse to the light-passingregion.

The device may also include a second wing region extending from thedistal end of the sleeve body adjacent to the light-passing regionhaving a second near-IR light source configured to emit near-IR lightfrom the second wing region. The device may also include an electricalcontact on a proximal end of the sleeve body configured to applyelectrical energy to the near-IR light source. The device may alsoinclude a flexible circuit coupling the electrical contact to thenear-IR light source. Any of these devices may include a camera sensoroperably connected to a second wing extending from the distal end of thesleeve body adjacent to the light-passing region.

Described herein are methods and apparatuses for taking, using anddisplaying dental information including information extracted fromthree-dimensional (3D) volumetric models of a patient's dental arch. A3D volumetric model may include surface (e.g., color) information aswell as information on internal structure, such as near-infrared(near-IR) transparency values for internal structures including enameland dentin. In some variations, the 3D volumetric scan may include or bederived from one or more other scanning modalities, including, but notlimited to: optical coherence tomography (OCT), ultrasound (US),magnetic resonance imaging (MRI), X-ray, etc.

In particular, described herein are methods and user interfaces fordisplaying and manipulating (e.g., sectioning, marking, selectingsub-regions, etc.) 3D volumetric models. For example, methods andapparatuses for displaying images from 3D volumetric models areprovided, including methods for generating sections though the 3Dvolumetric model, methods for showing both surface and internalstructures, and methods for generating easy to interpret images from the3D volumetric models, such as pseudo-x-ray images.

Also described herein are methods and apparatuses for marking andtracking regions of interest from a 3D volumetric model of a patient'sdental arch. These methods may include automatically, manually orsemi-automatically (e.g., with user approval or input) identifying oneor more regions from within the 3D volumetric model to mark (includingsurface features and/or internal features of the dental arch); theseregions may be regions in which a caries, crack or other irregularityhas developed or may develop. Marked regions may be analyzed in greaterdetail, and may be tracked over time. Further, marked regions may modifythe manner in which subsequent scanning is performed, e.g., by scanningmarked regions at higher resolution. The regions of the volumetric modelmay correspond to one or more voxels, including contiguous voxelregions. These regions may be referred to herein as volumetric regions.

Also described herein are methods and apparatuses for using 3Dvolumetric models to improve or modify a dental procedure, includingmodifying treatment planning and/or modifying one or more dental device.For example, described herein are dental tools that include 3Dvolumetric scanning, or that may be operated in conjunction with 3Dvolumetric models (including robotic or automated control using 3Dvolumetric models). Methods of diagnosing one or more conditions (e.g.,dental conditions) using a 3D volumetric model, and particularly using3D volumetric models over time are also described.

A method of displaying images from a three-dimensional (3D) volumetricmodel of a patient's dental arch, the method comprising: collecting the3D volumetric model of the patient's dental arch, wherein the 3Dvolumetric model includes surface color and shade values andnear-infrared (near-IR) transparency values for internal structureswithin the dental arch; selecting, by a user, an orientation of a viewof the 3D volumetric model to display; generating a two-dimensional (2D)view into the 3D volumetric using the selected orientation, includingthe patient's dental arch including a weighted portion of the surfacecolor values and a weighted portion of the near-IR transparency of theinternal structures; and displaying the 2D view.

For example, described herein are methods of displaying images from athree-dimensional (3D) volumetric model of a patient's dental arch. Themethod may include: receiving the 3D volumetric model of the patient'sdental arch, wherein the 3D volumetric model includes surface colorvalues and near-infrared (near-IR) transparency values for internalstructures within the dental arch; generating a two-dimensional (2D)view through the 3D volumetric model including the patient's dental archincluding both surface color values and the near-IR transparency of theinternal structures. In any of the methods and apparatuses describedherein, a 3D model (including a volumetric 3D model) may be displayed asa voxel view. Thus, the methods described herein may generate one ormore voxel views in which each voxel may have a color (or hue) thatcorresponds to its density and/or translucently. Thus, an of the methodsand apparatuses described herein may generate a 3D color map of all orsome of the voxels of the 3D model (and display one or more 2D imagesderived from the 3D color view, such a sections, slices, projections,perspective views, transparent-views in which all or some of the 3Dmodel is rendered transparent, etc.). In some variations, flaggedregions (e.g., regions corresponding to one or more irregular regions,and/or regions, e.g., voxels that have changed over time, regions/voxelsthat should be removed, regions/voxels suspected to be problematic andetc., may be displayed as a 3D and/or 2D view.

Generating the two-dimensional (2D) view through the 3D volumetric mayinclude: including in the 2D view, a weighted portion of the surfacecolor values and a weighted portion of the near-IR transparency of theinternal structures. Note that the near-IR transparency may be based onor otherwise calculated from near IR scattering or absorption of thematerial. The weighted portion of the surface color values may comprisea percentage of the full value of the surface color values, and theweighted portion of the near-IR transparency of the internal structurescomprises a percentage of the full value of the near-IR transparency ofthe internal structures, wherein the percentage of the full value of thesurface color values and the percentage of the full value of the near-IRtransparency of the internal structures adds up to 100%.

In some variations, the method also includes adjusting, by a user, or inresponse to user input, the weighted portion of the surface color valuesand/or the near-IR transparency of the internal structures.

Any of these methods may include the step of scanning the patient'sdental arch with an intraoral scanner.

Generating the 2D view may comprise sectioning the 3D volumetric modelin a plane through the 3D volumetric model. The user may select asection though the 3D volumetric model to display, and/or an orientationof the 2D view.

For example, a method of displaying images from a three-dimensional (3D)volumetric model of a patient's dental arch may include: receiving the3D volumetric model of the patient's dental arch, wherein the 3Dvolumetric model includes surface color values and near-infrared(near-IR) transparency values for internal structures within the dentalarch; selecting, by a user or in response to user input, a sectionthough the 3D volumetric model to display; generating a two-dimensional(2D) view through the 3D volumetric using the selected section,including the patient's dental arch, and possibly also including aweighted portion of the surface color values and a weighted portion ofthe near-IR transparency of the internal structures; and displaying the2D view.

A method of displaying images from a three-dimensional (3D) volumetricmodel of a patient's dental arch may include: collecting the 3Dvolumetric model of the patient's dental arch, wherein the 3D volumetricmodel includes surface values and near-infrared (near-IR) transparencyvalues for internal structures within the dental arch; generating atwo-dimensional (2D) view into the 3D volumetric model including thepatient's dental arch including both surface values and the near-IRtransparency of the internal structures; and displaying the 2D view.

A method of tracking a region of a patient's dental arch over time mayinclude: receiving a first three-dimensional (3D) volumetric model ofthe patient's dental arch, wherein the 3D volumetric model includessurface color values and near-infrared (near-IR) transparency values forinternal structures within the dental arch; identifying a region withinthe 3D volumetric model to be marked; flagging the identified region;and displaying one or more images of the 3D volumetric model indicatingthe marked region.

For example, a method of tracking a region of a patient's dental archover time, the method comprising: collecting a first three-dimensional(3D) volumetric model of the patient's dental arch, wherein the 3Dvolumetric model includes surface values and near-infrared (near-IR)transparency values for internal structures within the dental arch;identifying a region of the 3D volumetric model; flagging the identifiedregion; collecting a second 3D volumetric model of the patient's dentalarch; and displaying one or more images marking, on the one or moreimages, a difference between the first 3D volumetric model and thesecond 3D volumetric model at the flagged region.

Identifying the region may comprise automatically identifying using aprocessor. For example, automatically identifying may compriseidentifying a region having a possible defects including: cracks andcaries. Identifying the region having a possible defect may comprisecomparing a near-IR transparency value of a region within the 3D modelto a threshold value. Automatically identifying may comprise identifyinga surface color value outside of a threshold range. Automaticallyidentifying may comprise segmenting the 3D volumetric model to identifyenamel regions and identifying regions having enamel thicknesses below athreshold value. Flagging the identified region may compriseautomatically flagging the identified regions. Flagging the identifiedregion may comprise manually confirming the identified region forflagging.

Any of these methods may include receiving a second 3D volumetric modelof the patient's dental arch and displaying a difference between thefirst 3D volumetric model and the second 3D volumetric model at themarked region.

Further, any of these methods may include pre-scanning or re-scanningthe patient's dental arch wherein the flagged region is scanned at ahigher resolution or in other scanning modalities than un-flaggedregions.

For example, a method of tracking a region of a patient's dental archover time may include: receiving a first three-dimensional (3D)volumetric model of the patient's dental arch, wherein the 3D volumetricmodel includes surface color values and near-infrared (near-IR)transparency values for internal structures within the dental arch;identifying, using an automatic process, a region within the 3Dvolumetric model to be marked; flagging the identified regions;receiving a second 3D volumetric model of the patient's dental arch; anddisplaying a difference between the first 3D volumetric model and thesecond 3D volumetric model at the marked region. In some instances, thesecond 3D volumetric model of the patient's dental arch may be from ascan of the patient at a subsequent visit to the dental practitioner'soffice at a later date.

Thus, a method of tracking a region of a patient's dental arch over timemay include: collecting a first three-dimensional (3D) volumetric modelof the patient's dental arch taken at a first time, wherein the 3Dvolumetric model includes surface color values and near-infrared(near-IR) transparency values for internal structures within the dentalarch; identifying, using an automatic process, a region within the 3Dvolumetric model to be flagged; flagging the identified regions;collecting a second 3D volumetric model of the patient's dental archtaken at a separate time; and displaying a difference between the first3D volumetric model and the second 3D volumetric model at the flaggedregion.

Also described herein are methods of displaying pseudo x-ray images froma three-dimensional (3D) volumetric model of a patient's dental arch.For example, a method may include: receiving the 3D volumetric model ofthe patient's dental arch, wherein the 3D volumetric model includesnear-infrared (near-IR) transparency values for internal structureswithin the dental arch; generating a two-dimensional (2D) view throughthe 3D volumetric including the patient's dental arch including thenear-IR transparency of the internal structures; mapping the near-IRtransparency of the internal structures in the 2D view to a pseudo-X-raydensity in which the near-IR transparency values are inverted in value;and displaying the mapped pseudo-X-ray density. Generating the 2D viewmay comprise sectioning the 3D volumetric model in a plane through the3D volumetric model. The 3D volumetric model may include surfaceinformation.

For example, a method of displaying pseudo x-ray images from athree-dimensional (3D) volumetric model of a patient's dental arch mayinclude: collecting the 3D volumetric model of the patient's dentalarch, wherein the 3D volumetric model includes near-infrared (near-IR)transparency values for internal structures within the dental arch;generating a two-dimensional (2D) view into the 3D volumetric modelincluding the patient's dental arch including the near-IR transparencyof the internal structures; mapping the near-IR transparency of theinternal structures in the 2D view to a pseudo-X-ray density in whichthe pseudo-X-ray density values in the 2D view are based on the near-IRtransparency values that are inverted in value; and displaying themapped pseudo-X-ray density.

Any of these methods may include identifying a sub-region from the 3Dvolumetric model prior to generating the 2D view, wherein the 2D viewcomprises a 2D view of the identified sub-region. The method may alsoinclude segmenting the 3D volumetric model into a plurality of teeth,wherein generating the 2D view may comprise a 2D view including just oneof the identified teeth.

Mapping the near-IR transparency may include inverting the near-IRtransparency values so that enamel within the 2D view is brighter thandentin within the 2D view.

A method of displaying pseudo x-ray images from a three-dimensional (3D)volumetric model of a patient's dental arch may include: receiving the3D volumetric model of the patient's dental arch, wherein the 3Dvolumetric model includes surface features and near-infrared (near-IR)transparency values for internal structures within the dental arch inwhich enamel is more transparent than dentin; generating atwo-dimensional (2D) view through the 3D volumetric including thepatient's dental arch including the near-IR transparency of the internalstructures including dentin and enamel; mapping the near-IR transparencyof the internal structures in the 2D view to a pseudo-X-ray density inwhich the near-IR transparency values are inverted in value so that theenamel is brighter than the dentin; and displaying the mappedpseudo-X-ray density.

For example, a method of displaying pseudo x-ray images from athree-dimensional (3D) volumetric model of a patient's dental arch mayinclude: collecting the 3D volumetric model of the patient's dentalarch, wherein the 3D volumetric model includes surface features andnear-infrared (near-IR) transparency values for internal structureswithin the dental arch in which enamel is more transparent than dentin;generating a two-dimensional (2D) view into the 3D volumetric includingthe patient's dental arch including the near-IR transparency of theinternal structures including dentin and enamel; mapping the near-IRtransparency of the internal structures in the 2D view to a pseudo-X-raydensity in which the near-IR transparency values are inverted in valueso that the enamel is brighter than the dentin; and displaying themapped pseudo-X-ray density.

Also described herein are methods and apparatuses for virtuallyreviewing (e.g., virtually sectioning, virtually scanning, virtuallyexamining), in real time, a volumetric model of the patient's dentalarch(s). These apparatuses may include non-transitory, machine-readabletangible medium storing instructions for causing one or more machines toexecute operations for performing any of the methods described herein.In particular, any of these methods and apparatuses may operate on adata set that includes both a 3D model of the patient's dental arch, orin some variations, both of the patient's dental arches. The 3D modelmay be, but is not limited to, a 3D volumetric model; in some variationthe 3D model is a 3D surface model of the arch. This data set may alsoinclude a plurality of images of the dental arch, taken from differentpositions relative to the dental arch, such as different angles betweenthe plane of the image and the dental arch and different sub-regions ofthe dental arch. Some of these images may be taken from the occlusalsurface, some from the gingival side, and some from the lingual side. Insome variations the images may be the same (or a subset of) the imagesused to form the 3D model of the teeth. The data set may includemultiple images taken from the same, or nearly the same, region of thedental arch and angle relative to the dental arch. In some variations,the data set may include sets of two or more images (e.g., pairs ofimages) each taken at approximately the same region of the dental archand at the same angle relative to the dental arch but using differentimaging techniques (e.g., different imaging techniques, such as visiblelight, IR/near-IR, florescence, X-ray, ultrasound, etc.).

For example, a method may include: displaying a three-dimensional (3D)model of a patient's dental arch; displaying a viewing window over atleast a portion of the 3D model of the patient's dental arch; allowing auser to change a relative position between the viewing window and the 3Dmodel of the patient's dental arch; and continuously, as the userchanges the relative positions between the viewing window and the 3Dmodel of the patient's dental arch: identifying, from both the 3D modelof the patient's dental arch and a plurality of images of a patient'sdental arch taken from different angles and positions relative to thepatient's dental arch, an image taken at an angle and position thatapproximates a relative angle and position between the viewing windowrelative and the 3D model of the patient's dental arch; and displayingthe identified image taken at the angle and position that approximatesthe angle and position between the viewing window relative to the 3Dmodel of the patient's dental arch.

Any of the methods described herein, a data set may include the 3D modelof the patient's dental arch and a plurality of images of a patient'sdental arch taken from different angles and positions relative to thepatient's dental arch. A data set may also or alternatively includesmetadata associated with each (or each set) of the figures indicatingthe angle and/or region of the dental arch at which the image was taken.Additional metadata may be included (e.g., indicating a distance fromthe dental arch, indicating exposure time, indicating that the image isan average of other images, a quality metric for the image, etc.).

For example, described herein are methods for displaying a 3D model(e.g., surface 3D model) of the patient's teeth and/or volumetric modelof the patient's teeth) that a user can virtually scan in greater detailby moving a viewing window over the 3D model of the dental arch. Forexample, described herein are methods including: displaying athree-dimensional (3D) model of a patient's dental arch; displaying aviewing window over a portion of the 3D model of the patient's dentalarch; allowing a user to change a relative position between the viewingwindow and the 3D model of the patient's dental arch, including one ormore of: an angle between a plane of the viewing window and thepatient's dental arch, and a portion of the dental arch adjacent to theviewing window; and continuously, as the user changes the relativepositions between the viewing window and the 3D model of the patient'sdental arch: identifying, from both the 3D model of the patient's dentalarch and a plurality of images of a patient's dental arch (e.g., in somevariations from a data set comprising both the 3D model of the patient'sdental arch and a plurality of images of a patient's dental arch),wherein each image is taken from a different angle and position relativeto the patient's dental arch, an image taken at an angle and positionthat approximates the relative angle and position between the viewingwindow relative and the 3D model of the patient's dental arch; anddisplaying the identified image taken at the angle and position thatapproximates the angle and position of the viewing window relative tothe displayed 3D model of the patient's dental arch.

For example, a method may include: displaying a three-dimensional (3D)model of a patient's dental arch; displaying a viewing window over aportion of the 3D model of the patient's dental arch; allowing a user tochange a relative position between the viewing window and the 3D modelof the patient's dental arch, including one or more of: an angle betweenthe patient's dental arch relative and a plane of the viewing window,and a portion of the dental arch adjacent to the viewing window; andcontinuously, as the user changes the relative position between theviewing window and the 3D model of the patient's dental arch:identifying, from both the 3D model of the patient's dental arch and aplurality of pairs of images of a patient's dental arch (e.g.,optionally from a data set comprising both the 3D model of the patient'sdental arch and a plurality of images of a patient's dental arch),wherein each pair of the plurality of pairs includes a first imagingwavelength and a second imaging wavelength each taken at the same angleand position relative to the patient's dental arch, a pair of imagestaken at an angle and position that approximate the angle and positionof the viewing window relative to the displayed 3D model of thepatient's dental arch; and displaying at least one of the identifiedpair of images taken at the angle and position that approximate theangle and position of the viewing window relative to the displayed 3Dmodel of the patient's dental arch.

The methods and apparatuses described herein can be used with a 3D modelthat is a surface model or any representation of the patient's dentalarch(s). It may be, but does not have to be, a 3D volumetric model ofthe patient's teeth, e.g., constructed from images (e.g., the pluralityof images of a patient's dental arch taken from different angles andpositions relative to the patient's dental arch). The model may berepresentative of the patient's actual dentition, abstracted from thepatient's dentition, or generic.

As described herein, a method may include: displaying athree-dimensional (3D) model of a patient's dental arch; displaying aviewing window over a portion of the 3D model of the patient's dentalarch; allowing a user to change a relative position between the viewingwindow and the 3D model of the patient's dental arch, including one ormore of: an angle between the viewing window and the patient's dentalarch, and a portion of the dental arch adjacent to the viewing window;and continuously, as the user changes the relative positions between theviewing window and the 3D model of the patient's dental arch:identifying, both the 3D model of the patient's dental arch and aplurality of near-IR images of a patient's dental arch (e.g., from adata set comprising both the 3D model of the patient's dental arch and aplurality of images of a patient's dental arch), wherein each near-IRimage is taken from a different angle and position relative to thepatient's dental arch, a near-IR image taken at an angle and positionthat approximates the relative angle and position between the viewingwindow relative and the 3D model of the patient's dental arch; anddisplaying the identified near-IR image taken at the angle and positionthat approximates the angle and position of the viewing window relativeto the displayed 3D model of the patient's dental arch.

In any of these examples, the images may be images taken with apenetrating modality, such as with a near-IR. For example, describedherein are methods including: displaying a three-dimensional (3D) modelof a patient's dental arch; displaying a viewing window over a portionof the 3D model of the patient's dental arch; allowing a user to changea relative position between the viewing window and the 3D model of thepatient's dental arch, including one or more of: an angle between theviewing window and the patient's dental arch, and a portion of thedental arch adjacent to the viewing window; and continuously, as theuser changes the relative positions between the viewing window and the3D model of the patient's dental arch: identifying, from a data setcomprising both the 3D model of the patient's dental arch and aplurality of near-IR images of a patient's dental arch, wherein eachnear-IR image is taken from a different angle and position relative tothe patient's dental arch, a near-IR image taken at an angle andposition that approximates the relative angle and position between theviewing window relative and the 3D model of the patient's dental arch;and displaying the identified near-IR image taken at the angle andposition that approximates the angle and position of the viewing windowrelative to the displayed 3D model of the patient's dental arch.

Any of these methods may also include identifying and displayingmultiple images taken at the same angle and position relative to thedental arch. For example, the images may be both a visible light imageand a penetrative image (such as an IR/near-IR image, etc.). Forexample, described herein are: methods comprising: displaying athree-dimensional (3D) model of a patient's dental arch; displaying aviewing window over a portion of the 3D model of the patient's dentalarch; allowing a user to change a relative position between the viewingwindow and the 3D model of the patient's dental arch, including one ormore of: an angle between the patient's dental arch relative and a planeof the viewing window, and a portion of the dental arch adjacent to theviewing window; and continuously, as the user changes the relativeposition between the viewing window and the 3D model of the patient'sdental arch: identifying, from a data set comprising both the 3D modelof the patient's dental arch and a plurality of pairs of images of apatient's dental arch, wherein each pair of the plurality of pairsincludes a first imaging wavelength and a second imaging wavelength eachtaken at the same angle and position relative to the patient's dentalarch, a pair of images taken at an angle and position that approximatethe angle and position of the viewing window relative to the displayed3D model of the patient's dental arch; and displaying the identifiedpair of images taken at the angle and position that approximate theangle and position of the viewing window relative to the displayed 3Dmodel of the patient's dental arch.

In any of these methods, identifying may comprise determining aplurality images that approximate the relative angle and positionbetween the viewing window relative and the 3D model of the patient'sdental arch and averaging the plurality to form the identified image.For example, there may be multiple images in the data set taken atapproximately (e.g., within +/−0.1%, 0.5%, 1%, 2%, 3%, 4%, 5%, 7%, 10%,15%, 20%, etc.) of the same angle and approximately (e.g., within+/−0.1%, 0.5%, 1%, 2%, 3%, 4%, 5%, 7%, 10%, 15%, 20%, etc.) of the sameregion of the dental arch; these similar images may be combined to forman average image that may be better than the individual images.

In general, identifying one or more images taken at an angle andposition that approximates the relative angle and position between theviewing window relative and the 3D model of the patient's dental archmay be identifying within an acceptable spatial range. For example, animage that was taken at between +/−a few degrees of the same angle(e.g., +/−0.1 degree, 0.2 degree, 0.3 degrees, 0.4 degrees, 0.5 degrees,0.6 degrees, 1 degree, 1.2 degrees, 1.5 degrees, 1.7 degrees, 1.8degrees, 2 degrees, 2.2 degrees, 2.5 degrees, 3 degrees, 3.2 degrees,3.5 degrees, 4 degrees, 5 degrees, etc.) as the plane of the viewingwidow and within +/−a range of distance of the dental arch region overwhich the viewing window is positioned (e.g., +/−0.1 mm, 0.2 mm, 0.3 mm,0.4 mm, 0.5 mm, 0.6 mm, 0.7 mm, 0.8 mm, 0.9 mm, 1 mm, 1.1 mm, 1.2 mm,1.5 mm, 1.7 mm, 2.0 mm, 2.2 mm, 2.5 mm, etc.).

Any of these methods may include receiving, in a processor, the dataset. The data set may be received directly from an intraoral scanner,and/or stored and retrieved. In some variations the data set may betransmitted and received by the processor, in some variations theprocessor may read the data set from a memory (e.g., a data store)connected to the processor.

In general, any of these methods may include displaying the viewingwindow over a portion of the 3D model of the patient's dental arch. Theviewing window may be any shape or size, such as a circle, oval,triangle, rectangle, or other polygon. For example, the viewing windowmay be a loop through which the portion of the 3D model of the patient'sdental arch may be viewed. The viewing angle may allow the dental archto be visualized through at least a portion of the viewing window. Theviewing window may be smaller than the dental arch. In some variationsthe viewing window may be made larger or smaller by the user.

Typically these methods may include displaying via a user interface. Forexample, the user interface may display on a screen or screens thedental arch 3D model, the viewing window, and/or the image(s)corresponding to the view thorough the viewing window of the dentalarch. The user may (e.g., by manipulating the user interface, e.g., viaa control such as a mouse, keyboard, touchscreen, etc.) move the viewingwindow and dental arch independently. This movement, and the image(s)determined to correspond to the image though the viewing window of theregion and angle of the viewing window relative to the dental arch, maybe displayed in real time, as the user moves the viewing window and/ordental arch relative to each other.

For example, allowing the user to change the relative position betweenthe viewing window and the 3D model of the patient's dental arch mayinclude separately controlling the angle and/or rotation of the 3D modelof a patient's dental arch and the portion of the dental arch adjacentto the viewing window. In some variations, allowing the user to changethe relative position between the viewing window and the 3D model of thepatient's dental arch may comprise allowing the user to move the viewingwindow over the 3D model of the dental arch.

As mentioned, any of the images identified to as taken from an angle andposition corresponding to the angle and position of the viewing windowas it is moved over and/or around the dental arch (or as the dental archis moved relative to the viewing window) may be any one or moremodalities. Thus, for example, identifying an image that approximatesthe relative angle and position between the viewing window relative andthe 3D model of the patient's dental arch may include identifying oneof: a visible light image, an infrared image, and a florescent image.

Displaying the identified image(s) that approximates the angle andposition of the viewing window relative to the displayed 3D model maycomprise displaying the identified image in a window adjacent oroverlapping with the display of the 3D model of the patient's dentalarch. For example, the images may be displayed on a screen alongside the3D model of the dental arch; a the user moves the dental arch and/orimaging window, the image(s) may be shown in one or more windowschanging in real time or near real-time to reflect the relative positionof the 3D model of the dental arch and the viewing window.

Also described herein are non-transitory, machine-readable tangiblemedium storing instructions for causing one or more machines to executeoperations for performing any of the methods described herein, includingvirtually reviewing a patient's dental arch. For example, anon-transitory, machine-readable tangible medium may store instructionsfor causing one or more machines to execute operations for virtuallyreviewing a patient's dental arch including: displaying athree-dimensional (3D) model of a patient's dental arch; displaying aviewing window over a portion of the 3D model of the patient's dentalarch; allowing a user to change a relative position between the viewingwindow and the 3D model of the patient's dental arch, including one ormore of: an angle between the viewing window and the patient's dentalarch, and a portion of the dental arch adjacent to the viewing window;and continuously, as the user changes the relative positions between theviewing window and the 3D model of the patient's dental arch:identifying, from a data set comprising both the 3D model of thepatient's dental arch and a plurality of images of a patient's dentalarch, wherein each image is taken from a different angle and positionrelative to the patient's dental arch, an image taken at an angle andposition that approximates the relative angle and position between theviewing window relative and the 3D model of the patient's dental arch;and displaying the identified image taken at the angle and position thatapproximates the angle and position of the viewing window relative tothe displayed 3D model of the patient's dental arch.

For example, a non-transitory, machine-readable tangible medium storinginstructions for causing one or more machines to execute operations forvirtually reviewing a patient's dental arch, comprising: displaying athree-dimensional (3D) model of a patient's dental arch; displaying aviewing window over a portion of the 3D model of the patient's dentalarch; allowing a user to change a relative position between the viewingwindow and the 3D model of the patient's dental arch, including one ormore of: an angle between the viewing window and the patient's dentalarch, and a portion of the dental arch adjacent to the viewing window;and continuously, as the user changes the relative positions between theviewing window and the 3D model of the patient's dental arch:identifying, from a data set comprising both the 3D model of thepatient's dental arch and a plurality of images of a patient's dentalarch, wherein each image is taken from a different angle and positionrelative to the patient's dental arch, an image taken at an angle andposition that approximates the relative angle and position between theviewing window relative and the 3D model of the patient's dental arch;and displaying the identified image taken at the angle and position thatapproximates the angle and position of the viewing window relative tothe displayed 3D model of the patient's dental arch.

Also described herein are intraoral scanning systems that are configuredto perform the methods described herein. For example, an intraoralscanning system may include a hand-held wand having at least one imagesensor and a light source configured to emit light at a spectral rangewithin near-infrared (near-IR) range of wavelengths; a display output(e.g., a visual output such as a monitor, screen, virtual realityinterface/augmented reality interface, etc.); a user input device (e.g.,any control for receiving and transmitting user input, such as, but notlimited to: a keyboard, button, joystick, touchscreen, etc. The displayoutput and the user input device may be the same touchscreen); and oneor more processors operably connected to the hand-held wand, display anduser input device, the one or more processors configured to: display athree-dimensional (3D) model of a patient's dental arch on the displayoutput; display a viewing window over a portion of the 3D model of thepatient's dental arch on the display output; change a relative positionbetween the viewing window and the 3D model of the patient's dental archbased on input from the user input device; identify, from both the 3Dmodel of the patient's dental arch and a plurality of images of thepatient's dental arch taken from different angles and positions relativeto the patient's dental arch, a near-infrared (near-IR) image taken atan angle and position that approximates a relative angle and positionbetween the viewing window relative and the 3D model of the patient'sdental arch; and display the identified near-IR image taken at the angleand position that approximates the angle and position between theviewing window relative to the 3D model of the patient's dental arch.

The one or more processors of the intraoral scanning system may beconfigured to receive the plurality of images of the patient's dentalarch taken from different angles and positions relative to the patient'sdental arch. For example, the images may be taken by the image sensor(s)on the hand-held wand and transmitted to the one or more processorsand/or stored in a memory that is accessed by the one or moreprocessors. The system may also include a controller coordinating theactivity of the one or more processors, the wand, and the display output(and user input device). The controller may display the images and/or a3D model constructed from the images as a user operates the hand-heldwant to take images at different locations and/or angles relative to thepatient's dental arch(es).

The one or more processors may be configured to continuously identifythe near-IR image and display the identified near-IR image as the userchanges the relative positions between the viewing window and the 3Dmodel of the patient's dental arch. Thus, as the user (using the userinput) adjusts the position of the viewing window (e.g., loop) relativeto the 3D model of the patient's dental arch on the display output (or,equivalently, adjusts the position of the 3D model of the dental arch onthe display output relative to the viewing window), the one or moreprocessors may determine and display a near-IR image of the patient'steeth that most closely approximates the relative positions between theviewing window and the 3D model of the patient's dental arch.

The near-IR image is either one of the images taken by the hand-heldwand or an average of the images taken by the hand-held wand. Any of theapparatuses (e.g., intraoral scanning systems) described herein may alsodetermine and/or store the positions and/or orientation of the hand-heldwand as it is being operated, and this information may be stored withthe image(s) taken from this position. For example, the hand-held wandmay include one or more accelerometers. For example, the one or moreprocessors may be configured to identify the near-IR image taken at anangle and position that approximates a relative angle and positionbetween the viewing window relative and the 3D model of the patient'sdental arch by determining a plurality images that approximate therelative angle and position between the viewing window relative and the3D model of the patient's dental arch and averaging the plurality toform the identified near-IR image.

As mentioned, the one or more processors may be configured to change, onthe display output, the relative position between the viewing window andthe 3D model of the patient's dental arch based on input from the userinput device. Specifically, the one or more processor may be configuredto change, based on user input into user input device, one or more of:an angle between a plane of the viewing window and the patient's dentalarch, and a portion of the dental arch adjacent to the viewing window(e.g., in some variations, visible through the viewing window). Asdiscussed above, the viewing window may be a loop (e.g., circular, oval,square, etc.) through which the 3D model is visible). Thus, the one ormore processors may be configured to display the viewing window over aportion of the 3D model of the patient's dental arch comprisesdisplaying as a loop through which the portion of the 3D model of thepatient's dental arch may be viewed. The viewing window may be moved andpositioned over (including changing which side of the dental arch(buccal, occlusal, lingual, or between these, including moving in x, y,z and/or in rotation, e.g., pitch, roll, yaw) the viewing window ispositioned over and/or the 3D model of the patient's teeth may be moved(e.g., rotating in pitch, yaw, roll, moving in x, y, z, etc.). Thus, theone or more processors may be configured to change the relative positionbetween the viewing window and the 3D model of the patient's dental archbased on input from the user input device by changing one or more of:the angle of the 3D model of a patient's dental arch relative to theviewing window (which is equivalent to the angle of the viewing windowrelative to the 3D model of the patient's dental arch), the rotation ofthe 3D model of a patient's dental arch relative to the viewing window(which is equivalent to the rotation of the viewing window relative tothe 3D model of a patient's dental arch), and the portion of the dentalarch adjacent to the viewing window (e.g., the portion of the 3D modelvisible through the viewing window). For example, the one or moreprocessors may be configured to change the relative position between theviewing window and the 3D model of the patient's dental arch based oninput from the user input device by changing the position of the viewingwindow over the 3D model of the dental arch.

The one or more processors may be configured to identify from both the3D model of the patient's dental arch and the plurality of images of thepatient's dental arch taken from different angles and positions relativeto the patient's dental arch, a second image that approximates therelative angle and position between the viewing window relative and the3D model of the patient's dental arch that is one or more of: a visiblelight image and a florescent image; and wherein the one or moreprocessors is configured to display the second image concurrently withthe near-IR image.

Also described herein are methods of automatically,semi-automatically/semi-manually or manually identifying and gradingfeatures by coordinating across multiple imaging modalities. Forexample, a dental diagnostic method may include: identifying a dentalfeature in a first record, the first record comprising a plurality ofimages of a patient's dental arch taken first imaging modality;correlating the first record with a model of the patient's dental arch;identifying, using the model of the patient's dental arch, a region ofthe dental arch corresponding to the dental feature in one or moredifferent records, wherein each record of the one or more differentrecords is taken with a different imaging modality than the firstimaging modality and wherein each of the one or more different recordsis correlated with the model of the patient's dental arch; determining aconfidence score for the dental feature based on the identified regionscorresponding to the dental feature in the one or more differentrecords; and displaying the dental feature when the confidence score forthe dental feature is above a threshold.

A dental diagnostic method may include: identifying a dental feature ina first record, the first record comprising a plurality of images of apatient's dental arch taken first imaging modality; correlating thefirst record with a three-dimensional (3D) volumetric model of thepatient's dental arch; flagging the dental feature on the 3D volumetricmodel; identifying, using the model of the patient's dental arch, aregion of the dental arch corresponding to the dental feature in one ormore different records, wherein each record of the one or more differentrecords is taken with a different imaging modality than the firstimaging modality and wherein each of the one or more different recordsis correlated with the model of the patient's dental arch; determiningor adjusting a confidence score for the dental feature based on theidentified regions corresponding to the dental feature in the one ormore different records; and displaying the dental feature and anindicator of the confidence score for the dental feature when theconfidence score for the dental feature is above a threshold.

In any of these methods (or systems for performing them) the dentalfeature may comprise one or more of: cracks, gum recess, tartar, enamelthickness, pits, caries, pits, fissures, evidence of grinding, andinterproximal voids.

Displaying may comprise displaying the dental feature and an indicatorof the confidence score for the dental feature.

Correlating the first record with the model of the patient's dental archmay comprise correlating the first record with a three-dimensional (3D)volumetric model of the patient's dental arch. Any of these methods (orsystems for performing them) may include flagging the dental feature onthe model of the patient's dental arch, and/or collecting the dentalfeature, including the location of the dental feature, and one or moreof: the type of dental feature and a confidence score for the dentalfeature.

Determining the confidence score may comprise adjusting the confidencescore for the dental feature based on the identified regionscorresponding to the dental feature in the one or more differentrecords.

In any of these methods or systems, identifying the dental feature maycomprise automatically identifying the dental feature.

For example, a dental diagnostic method may include: identifying one ormore actionable dental features from one or more records of a pluralityof records, wherein each record comprises a plurality of images of apatient's dental arch each taken using an imaging modality, furtherwherein each record of the plurality of records is taken at a differentimaging modality; mapping the actionable dental feature to acorresponding region of the one or more records; recording the one ormore actionable dental features, including recording a location of theactionable dental feature; adjusting or determining a confidence scorefor the one or more actionable dental features based on thecorresponding region of the one or more records; and displaying the oneor more actionable dental features when the confidence score of the oneor more actionable dental features is above a threshold. As mentionedabove, the one or more actionable dental feature comprises one or moreof: cracks, gum recess, tartar, enamel thickness, pits, caries, pits,fissures, evidence of grinding, and interproximal voids.

Displaying may comprise displaying the one or more actionable dentalfeatures and an indicator of the confidence score for the dentalfeature. Mapping the actionable dental feature to the correspondingregion of the one or more records may comprise correlating the firstrecord with a three-dimensional (3D) volumetric model of the patient'sdental arch. Recording the one or more actionable dental features maycomprise marking the dental feature on the 3D volumetric model of thepatient's dental arch. Identifying the dental feature may compriseautomatically identifying the dental feature.

Also described herein are systems for performing any of the methodsdescribed herein. For example, a system may include: one or moreprocessors; and a memory coupled to the one or more processors, thememory configured to store computer-program instructions, that, whenexecuted by the one or more processors, perform a computer-implementedmethod comprising: identifying a dental feature in a first record, thefirst record comprising a plurality of images of a patient's dental archtaken first imaging modality; correlating the first record with a modelof the patient's dental arch; identifying, using the model of thepatient's dental arch, a region of the dental arch corresponding to thedental feature in one or more different records, wherein each record ofthe one or more different records is taken with a different imagingmodality than the first imaging modality and wherein each of the one ormore different records is correlated with the model of the patient'sdental arch; determining a confidence score for the dental feature basedon the identified regions corresponding to the dental feature in the oneor more different records; and displaying the dental feature when theconfidence score for the dental feature is above a threshold.

Also described herein are methods and apparatuses (e.g., systems) fortracking one or more regions (e.g., tagged or flagged regions) acrossdifferent imaging modalities and/or over time. For example, a method oftracking a dental feature across different imaging modalities mayinclude: collecting a first three-dimensional (3D) volumetric model ofthe patient's dental arch, wherein the 3D volumetric model of thepatient's dental arch includes surface values and internal structureswithin the dental arch; identifying a region of the patient's dentalarch from a first record of a plurality of records, wherein each recordcomprises a plurality of images of a patient's dental arch each takenusing an imaging modality, further wherein each record of the pluralityof records is taken at a different imaging modality; flagging theidentified region in a corresponding region of the 3D volumetric modelof the patient's dental arch; correlating the flagged region with eachof records of the plurality of records by correlating the 3D volumetricmodel of the patient's dental arch with each of the records of theplurality of records; and saving, displaying and/or transmitting imagesincluding the region of the patient's dental arch. The region of thepatient's dental arch may comprise a dental feature comprises one ormore of: cracks, gum recess, tartar, enamel thickness, pits, caries,pits, fissures, evidence of grinding, and interproximal voids.

Saving, displaying and/or transmitting may comprise displaying theregions of the patient's dental arch. Any of these methods may includeflagging the dental feature on the 3D volumetric model. Identifying theregion of the patient's dental arch may comprise automaticallyidentifying the region of the patient's dental arch.

A system for tracking one or more regions (e.g., tagged or flaggedregions) across different imaging modalities and/or over time mayinclude: one or more processors; a memory coupled to the one or moreprocessors, the memory configured to store computer-programinstructions, that, when executed by the one or more processors, performa computer-implemented method comprising: collecting a firstthree-dimensional (3D) volumetric model of the patient's dental arch,wherein the 3D volumetric model of the patient's dental arch includessurface values and internal structures within the dental arch;identifying a region of the patient's dental arch from a first record ofa plurality of records, wherein each record comprises a plurality ofimages of a patient's dental arch each taken using an imaging modality,further wherein each record of the plurality of records is taken at adifferent imaging modality; flagging the identified region in acorresponding region of the 3D volumetric model of the patient's dentalarch; correlating the flagged region with each of records of theplurality of records by correlating the 3D volumetric model of thepatient's dental arch with each of the records of the plurality ofrecords; and saving, displaying and/or transmitting images including theregion of the patient's dental arch.

BRIEF DESCRIPTION OF THE DRAWINGS

The novel features of the invention are set forth with particularity inthe claims that follow. A better understanding of the features andadvantages of the present invention will be obtained by reference to thefollowing detailed description that sets forth illustrative embodiments,in which the principles of the invention are utilized, and theaccompanying drawings of which:

FIG. 1A illustrates one example of a 3D (color) intraoral scanner thatmay be adapted for used as described herein to generate a model ofsubject's teeth having both surface and internal features.

FIG. 1B schematically illustrates an example of an intraoral scannerconfigured to generate a model of subject's teeth having both surfaceand internal features.

FIG. 2A illustrates trans-illumination imaging through a tooth at 180°.

FIG. 2B illustrates trans-illumination imaging through a tooth at 90°.

FIGS. 2C and 2D show side and top perspective views, respectively, of anexample of a distal end of a wand of an intraoral scanner configured toprovide trans-illumination imaging through a tooth at 90° and 180°.

FIG. 2E shows a schematic of an intraoral scanner configured to do bothsurface scanning (e.g., visible light, non-penetrative) and penetrativescanning using a near infra-red (IR) wavelength. The scanner includes apolarizer and filters to block near-IR light reflected off the surfaceof the tooth while still collecting near-IR light reflected frominternal structures.

FIGS. 3A, 3B and 3C illustrate exemplary penetration with small angleillumination imaging orientations using an intraoral scanner wand suchas the one shown in FIGS. 2C and 2D.

FIG. 3D shows another example (similar to what is shown in FIG. 3A) ofilluminating with light (e.g., near-IR) on the right side and imagingfrom the left side (this orientation may be flipped) to get 180°trans-illumination. Higher and lower scattering is shown by the arrows.

FIG. 4A illustrates an example of a penetration imaging (e.g.,small-angle penetration imaging) configuration of sensors and light(illumination) sources in which the viewing vector between the sensorsand light sources is between 0° and 15° at different positions around atooth; these different positions represent different positions taken atdifferent times, e.g., by moving the wand/scanner around the tooth sothat penetrative images may be taken at different angles relative to thetooth.

FIGS. 4B-4F illustrate other variations of penetrative imaging similarto that shown in FIG. 4A, for imaging from a tooth. FIG. 4B shows anexample of a multi-camera, multi-light source scanner. FIGS. 4C-4F showalternative small-angle configurations.

FIGS. 5A-5I illustrate nine alternative penetration imaging orientationsthat may be used as part of an intraoral scanner such as the ones shownin FIGS. 1A-1B. In FIGS. 5A-5C the central sensor is active, and eitherthe right (FIG. 5B) or the left (FIG. 5A) or both (FIG. 5C) lightsources are illuminating the tooth. Similarly, in FIGS. 5D-5E the rightsensor is active, while in FIGS. 5G-5I the left sensor is active.

FIG. 6 is a diagram schematically illustrating one method of generatinga model of a subject's tooth or teeth having both surface and internalfeatures.

FIG. 7 is a diagram illustrating one variation of a method of generatinga model of a subject's teeth when having both surface and internalfeatures by cycling between different scanning modalities (e.g., surfacescanning, penetration imaging, etc.).

FIG. 8 is a graphical example of one a timing diagram for scanning asample (e.g., tooth) to generate a model having both surface andinternal features cycling between different scanning modalities (showingsurface scanning, laser florescence, viewfinder and penetration imagingmodalities). In FIG. 8, the y-axis indicates the lens position of the 3Dconfocal scanner (scan amplitude). The durations of each of the scans(e.g., the scanning time for each mode) may be fixed, or it may beadjustable. For example the duration of the penetrative scan (d) may bedynamically adjusted (e.g., increased or decreased) during scanning baseon the quality of the images received, the completeness of the 3Dreconstruction of internal structures, etc. Similarly, the duration ofthe surface scan may be dynamically adjusted during scanning based onthe quality of the image(s) being scanned (e.g., the prior images and/orthe current image, etc.), the completeness of the 3D surface model forthe region being scanned, etc.

FIG. 9A illustrates one example of an overlay of the penetration imagesin on a 3D surface model of the teeth, showing the image penetrationpanorama (in which penetrating images are stitched together to form thepanorama)

FIG. 9B illustrates the portion of the model reconstruction of FIG. 9Aincluding the surface and internal features. Note that in FIGS. 9A and9B, the overlay showing the internal structures is not a volumetricreconstruction.

FIG. 10A shows an example of a front view of one example of an intraoralscanner front end.

FIG. 10B shows an example of a bottom view of the intraoral scanner,showing the plurality of sensors and light sources.

FIGS. 11A-11C shows projected images looking down through the top of thetooth using a penetrative wavelength (e.g., near-IR).

FIGS. 11D-11F illustrate the movement of the light source relative tothe tooth in the z direction, using a penetrative wavelength.

FIGS. 11G-11I show the position of the scanner, such as thoseillustrated above, scanning the tooth in the z-direction. Note thatFIGS. 11A, 11D and 11G correspond to a first depth position, FIGS. 11B,11E and 11H correspond to a second (higher up the tooth) depth position,and FIGS. 11C, 11F and 11I correspond to a third (even higher up) depth.

FIG. 12 illustrates an example of a configuration of penetration lightsources (e.g., penetrative spectral range light) and cameras that may beused as part of an intraoral scanner wand.

FIG. 13 shows a flowchart that describes one method for reconstructing avolumetric structure from an object including semi-transparent stronglyscattering regions for a range of radiation wavelengths.

FIG. 14 illustrates another flowchart that provides method steps forreconstructing a volumetric structure from a tooth.

FIGS. 15A-15E show one example of an image fixed pattern noisecalibration and illumination non-uniformity calibration, which gives aconstant response for a uniform plane target.

FIG. 16 is a simplified block diagram of a data processing system whichcan be used to perform the methods and techniques described herein.

FIG. 17 is an example of a method of scanning teeth with an intraoralscanner to identify internal structures using a penetrative wavelength(e.g., IR and/or near-IR).

FIGS. 18A-18C illustrate one method of automatic segmentation of anear-IR image. FIG. 18A illustrates edge detection from a penetrativescan through the teeth, taken with an intraoral scanner in the near-IRwavelength (e.g., 850 nm). FIGS. 18B and 18C shows segmentation based onthe edge detection of FIG. 18A plotted on the penetrative scan.

FIGS. 19A-19C show further segmentation of the near-IR image of FIGS.18A-18C. FIG. 19A shows edge detection from the near-IR image taken of asubject's teeth shown in FIG. 19C. FIG. 19B shows segmentation of theimage of FIG. 19C, in which the segments (5 segments) are drawn on thenear-IR image shown in FIG. 19C.

FIGS. 20A-20C illustrate segmentation of a near-IR image of a patient'steeth. FIG. 20A is a figure showing edge detection of a near-IR image.FIG. 20B illustrates segmentation of the near-IR image, showing 18(overlapping) segments. FIG. 20C illustrates further segmentation of thenear-IR image shown in FIG. 20B.

FIGS. 21A-21C illustrate segmentation of a near-IR image of a patient'steeth. FIG. 21A shows edge detection of the near-IR image of FIG. 21C.FIG. 21B illustrates edge detection of the near-IR image shown in FIG.21C.

FIGS. 22A-22C illustrate segmentation of a near-IR image of a patient'steeth. FIG. 22A is a figure showing edge detection of a near-IR image.FIG. 22B illustrates segmentation of the near-IR image, showing 8(overlapping) segments. FIG. 22C illustrates further segmentation of thenear-IR image shown in FIG. 22B.

FIGS. 23A-23C illustrate segmentation of a near-IR image of a patient'steeth. FIG. 23A shows edge detection of the near-IR image of FIG. 23C.FIG. 23B illustrates edge detection of the near-IR image shown in FIG.23C.

FIG. 24A is a partial three-dimensional model of a patient's teethformed by segmented images, including those shown in FIGS. 18A-23C.

FIG. 24B shows a sectional view through the 3D model of FIG. 24A,showing internal structures, including the dentin.

FIG. 25A is an example of a volumetric (or “voxel”) model of a patient'sjaw and teeth, including internal structures. The internal structuresare shown as a density map within the 3D surface model. FIG. 25B is anenlarged view of the volumetric model of FIG. 25A.

FIGS. 26A-26C illustrate a method of forming a 3D surface that may beused to generate a volumetric model (showing both surface and internalstructures) of a patient's teeth.

FIGS. 27A-27G illustrate a method of generating a volumetric model of apatient's teeth using near-IR scanning in addition to surface scanning.

FIGS. 28A and 28B illustrate volumetric models of a patient's teethformed using an intraoral scanner, showing both surface features, e.g.,enamel, and internal (segmented) features, e.g., dentin.

FIG. 29A shows a partially transparent perspective view of aremovable/disposable cover configured as a trans-illumination sleevewith electrical couplings. FIG. 29B is a perspective view of the sleeveof FIG. 29A, shown solid. This sleeve is configured for use with a wandportion of an intraoral scanner; the sleeve is configured to adapt thewand to include trans-illumination with a penetrative (e.g., near-IR)wavelength.

FIGS. 30A-30C illustrate one example of a trans-illumination sleeve withelectrical couplings. FIG. 30A shows an example of a supporting frame ofthe sleeve; FIG. 30B shows the support frame with a flex circuit andconnectors coupled to the supporting frame. FIG. 30C shows the fullyassembled sleeve of FIGS. 30A-30B.

FIG. 31A shows an example of a flex circuit and connectors for use aspart of the sleeve shown in FIGS. 29A-30B. FIG. 31B is an example of adistal end portion of the flex circuit shown in FIG. 31A, including anLED housing. FIG. 31C is an example of a connector portion of a sleeve.

FIGS. 32A and 32B illustrate examples of an LED positioner and lightblocker portion of the distal end of a sleeve such as the ones shown inFIGS. 29A-30B.

FIG. 33 illustrates one method of generating internal structure (orpseudo x-ray) images from a volumetric data.

FIGS. 34A and 34B illustrate virtual sections from a volumetric model ofthe teeth. These virtual sections may be annotated,colored/pseudo-colored, or textured, to show internal features orproperties of the teeth. In FIG. 34A, virtual section is pseudo-coloredto show enamel; in FIG. 34B, the virtual section is pseudo-colored toshow dentin.

FIG. 35 illustrates one method of marking (e.g., flagging) a volumetricmodel of a patient's teeth, and/or using the marked regions.

FIG. 36 is a comparison between a typical computer-aideddesign/computer-aided manufacturing (CAD/CAM) method for dentistry, anda method implementing the 3D volumetric scanning and modeling asdescribed herein.

FIG. 37A is an example of a display tracking gingival recession overtime in using a 3D volumetric model as described herein. FIG. 37B showsan enlarged view of region B in FIG. 37A showing the later time.

FIGS. 38A-38G illustrate one method of displaying volumetric informationfrom a patient's teeth. FIG. 38A show an example of a 3D volumetricmodel of a patient's upper jaw (showing teeth and gingiva), from a topview. FIG. 38B shows the same 3D volumetric model, showing the internalfeatures, including the more transparent enamel and the less transparentdentin. The 3D volumetric model may be manipulated to show more or lessof the surface and/or internal structures. FIGS. 38C-38G illustrateprogressively more transparent views or a region (“C”) of the 3Dvolumetric model of FIG. 38A. FIG. 38C show a 2D image extracted from aregion of the 3D volumetric model showing just the outer surface of theteeth (e.g., 100% of the color/outer surface image, 0% near-IR/internalvolume). FIG. 38D shows the same region as FIG. 38C, combining the outersurface (color) image and the internal (near-IR based) image (e.g., 75%of the color/outer surface image, 25% near-IR/internal volume). FIG. 38Eshows the same region as FIG. 38C, combining the outer surface (color)image and the internal (near-IR based) image (e.g., 50% of thecolor/outer surface image, 50% near-IR/internal volume). FIG. 38F showsthe same region as FIG. 38C, combining the outer surface (color) imageand the internal (near-IR based) image (e.g., 25% of the color/outersurface image, 75% near-IR/internal volume). FIG. 38G shows the sameregion as FIG. 38C showing just the internal (near IR based) image ofthe teeth (e.g., 0% of the color/outer surface image, 1000%near-IR/internal volume).

FIG. 39A illustrates an example of a user interface for analysis and/ordisplay of a 3D volumetric model of a patient's teeth, showing a topview of the upper arch, tools that may be used to manipulate theview(s), and two enlarged views showing the outer surface of an enlargedregion of the tooth (on the left) and the same view showing internalfeatures of the tooth (showing dentin and enamel within the tooth).

FIG. 39B show the user interface of FIG. 39A in which a region of theteeth has been marked/flagged as described herein.

FIGS. 40A-40C illustrate another example of a method of displaying 3Dvolumetric image information by mixing it with surface (non-penetrative)information. FIG. 40A shows a visible light image of a region of apatient's dental arch taken with a scanner that is also configured totake penetrative (near-IR) scans). FIG. 40B show a volumetric model ofthe reconstructed 3D volumetric model of a patient's tooth showinginternal dentin and enamel. Features not visible on the surface scan areapparent in the volumetric scan, including a caries and a bubbled regionwithin the enamel. FIG. 40C shows a hybrid image in which the 3Dvolumetric image has been combined with the surface scan, showing bothsurface and internal structures, including the carries and the bubbledregion.

FIG. 41 is an example of a method for allowing a user to virtually scana patient's dental arch. This method may be performed in real time ornear real time.

FIG. 42 is a schematic illustration of a data structure including a 3Dmodel of a patient's dental arch(s) and associated 2D images taken(e.g., via intraoral scanner) of the dental arch at a large number ofpositions around the dental arch.

FIG. 43A is an example of a user interface allowing the user tovirtually scan over the 3D model of the dental arch, showingcorresponding light and near-IR (e.g., external and internal) regions indetail as the user scans over the 3D dental arch; the user may use oneor more tools to move the dental arch (e.g., rotate, translate, etc.)and/or the viewing window; the corresponding light and near-IR imagesmay continuously or near-continuously update as the position of theviewing window and dental arch change. A pair of imaging windows areshown adjacent to the view of the 3D model of the dental arch.

FIG. 43B is an alternative display showing a single large image windowover or adjacent to the 3D image of the dental arch. In FIG. 43B theimage window show a light image of the corresponding region of thedental arch.

FIG. 43C is an alternative display showing a single large image windowover or adjacent to the 3D image of the dental arch. In FIG. 43C theimage window show a near-IR image of the corresponding region of thedental arch.

FIG. 44A is similar to FIG. 43A, showing an example of a 3D model of theouter surface of a dental arch, and a viewing window relative to thedental arch. A pair of image display windows are adjacent to the 3Dmodel of the dental arch. The user may move the viewing window over thedental arch (and/or may move the dental arch relative to the viewingwindow, changing the image(s) shown in the two display windows. Theupper display window shows a near-IR image corresponding to the dentalarch at the position and angle of the plane of the viewing window; thebottom display window shows a corresponding light image (which may be incolor).

FIG. 44B shows another image of the dental arch shown in FIG. 44A, withthe dental arch rotated lingually relative to the viewing window; thecorresponding near-IR images (upper right) and visible light (lowerright) adjacent to the 3D model of the arch are updated to show theslightly rotated view, allowing the user to virtually scan the dentalarch and show both external and internal views in real (or near-real)time.

FIG. 45A is another example of a method of showing a 3D model of adental arch (shown as the lower arch in this example, e.g., by selectingthe lower arch display control in the upper left of the user interface)and showing focused views of near-IR and visible light imagescorresponding to the viewing window region that may be movedover/across, and around (lingual-occlusal-buccal) the model of thepatient's arch.

FIG. 45B shows an example of a single window (an enlarged near-IR viewinto the teeth of the region corresponding to the viewing window loop)similar to FIG. 45A.

FIG. 45C shows an example of a single window (an enlarged visible lightview into the teeth of the region corresponding to the viewing windowloop) similar to FIG. 45A.

FIG. 46 schematically illustrates one example of a method forautomatically or semi-automatically identify, confirm and/orcharacterize one or more actionable dental features that may benefitfrom detection and/or treatment.

DETAILED DESCRIPTION

Described herein are intraoral scanners for generating athree-dimensional (3D) model of a subject's intraoral region (e.g.,tooth or teeth, gums, jaw, etc.) which may include internal features ofthe teeth and may also include a model of the surface, and methods ofusing such scanners. For example, FIG. 1A illustrates one example of anintraoral scanner 101 that may be configured or adapted as describedherein to generate 3D models having both surface and internal features.As shown schematically in FIG. 1B, an exemplary intraoral scanner mayinclude a handle or wand 103 that can be hand-held by an operator (e.g.,dentist, dental hygienist, technician, etc.) and moved over a subject'stooth or teeth to scan both surface and internal structures. The wandmay include one or more sensors 105 (e.g., cameras such as CMOS, CCDs,detectors, etc.) and one or more light sources 109, 110, 111. In FIG.1B, three light sources are shown: a first light source 109 configuredto emit light in a first spectral range for detection of surfacefeatures (e.g., visible light, monochromatic visible light, etc.; thislight does not have to be visible light), a second color light source(e.g., white light between 400-700 nm, e.g., approximately 400-600 nm),and a third light source 111 configured to emit light in a secondspectral range for detection of internal features within the tooth(e.g., by trans-illumination, small-angle penetration imaging, laserflorescence, etc., which may generically be referred to as penetrationimaging, e.g., in the near-IR). Although separate illumination sourcesare shown in FIG. 1B, in some variations a selectable light source maybe used. The light source may be any appropriate light source, includingLED, fiber optic, etc. The wand 103 may include one or more controls(buttons, switching, dials, touchscreens, etc.) to aid in control (e.g.,turning the wand on/of, etc.); alternatively or additionally, one ormore controls, not shown, may be present on other parts of the intraoralscanner, such as a foot petal, keyboard, console, touchscreen, etc.

In general, any appropriate light source may be used, in particular,light sources matched to the mode being detected. For example, any ofthese apparatuses may include a visible light source or other (includingnon-visible) light source for surface detection (e.g., at or around 680nm, or other appropriate wavelengths). A color light source, typically avisible light source (e.g., “white light” source of light) for colorimaging may also be included. In addition a penetrating light source forpenetration imaging (e.g., infrared, such as specifically near infraredlight source) may be included as well.

The intraoral scanner 101 may also include one or more processors,including linked processors or remote processors, for both controllingthe wand 103 operation, including coordinating the scanning and inreviewing and processing the scanning and generation of the 3D modelincluding surface and internal features. As shown in FIG. 1B the one ormore processors 113 may include or may be coupled with a memory 115 forstoring scanned data (surface data, internal feature data, etc.).Communications circuitry 117, including wireless or wired communicationscircuitry may also be included for communicating with components of thesystem (including the wand) or external components, including externalprocessors. For example the system may be configured to send and receivescans or 3D models. One or more additional outputs 119 may also beincluded for outputting or presenting information, including displayscreens, printers, etc. As mentioned, inputs 121 (buttons, touchscreens,etc.) may be included and the apparatus may allow or request user inputfor controlling scanning and other operations.

Any of the apparatuses and methods described herein may be used to scanfor and/or identify internal structures such as cracks, caries (decay)and lesions in the enamel and/or dentin. Thus, any of the apparatusesdescribed herein may be configured to perform scans that may be used todetect internal structures using a penetrative wavelength or spectralrange of penetrative wavelengths. Also described herein are methods fordetecting cracks, caries and/or lesions or other internal feature suchas dental fillings, etc. A variety of penetrative scanning techniques(penetration imaging) may be used or incorporated into the apparatus,including but not limited to trans-illumination and small-anglepenetration imaging, both of which detect the passage of penetrativewavelengths of light from or through the tissue (e.g., from or through atooth or teeth).

Trans-illumination is one technique that may be used for seeing internalfeatures of teeth. Traditionally, there are 2 basic configurations fortrans-illumination through the teeth. FIGS. 2A and 2B illustrate these:a 180° configuration and a 90° configuration. Both configurations may beused for visualizing inside the teeth, and mainly through the enamel. Asshown in FIG. 2A, in the 180° configuration, a penetrative wavelength(including a spectral range of one or more penetrative wavelengths) isemitted from a light source 203 and passed from one side of the tooth201, and a sensor 205 (e.g., camera) on the opposite side detects thelight that has passed through the tooth without being scattered orabsorbed. Similarly, in FIG. 2B, the tooth 201 is illuminated by lightfrom light sources (203, 203′) on either side of the tooth 201, and thecamera 205, which is oriented 90° relative to both light sources, detectlight at the right angle to the light source. Typically,trans-illumination has been limited to the use of a single projectiontype, in order to give an image capture inside the tooth (similar to theuse of an x-ray). Described herein are methods and apparatuses forvisualization of the enamel-dentin area using a penetrative wavelength(such as between 700 to 1300 nm, 700 to 1090 nm, etc., e.g., 850 nm) andacquiring a plurality of projections or orientations from a singleposition of the scanner relative to the tooth/teeth and/or for aplurality of angles of the sensor relative to the teeth; in particularthree or more orientations or projections may be taken for each internalregion being imaged. Taking multiple (e.g., 3 or more) projections mayprovide better imaging, as it may produce multiple (e.g., 3 or more)images through the tooth from a particular location of the wand relativeto the tooth/teeth. The use of one or more 180° projection may be usefulas the light travels a shorter distance and is less scattered, howeverthe combination of multiple different projections (orientations) fromthe same location (e.g., at approximately the same scanning time, withina few milliseconds of each other) may permit the system to build avolumetric model of the enamel-dentin area.

In variations using 90 and/or 180° configuration projections, theintraoral scanner may be adapted to provide trans-illumination imagingin this configuration. For example, FIGS. 2C and 2D illustrate oneexample of a distal end of a wand of an intraoral scanner adapted tocollect trans-illumination images at 90 and 180°, in which the wand 213includes a pair of projections or wings 215 each housing a light source(LED) and camera combination 217. This distal end portion of the wandmay be configured as a sleeve 214, as shown (additional and alternativeexamples of sleeves are shown in FIGS. 29A-30C, described below). Thesleeve may be removably attached onto and/or over the distal end of thewand. In FIGS. 2C and 2D, both wings and the base of the wand mayinclude light sources and sensors (cameras) so that at least threetrans-illumination images may be taken from a single position of thewand relative to the teeth, as shown in FIGS. 3A-3C. In FIG. 3A a firstorientation is shown, in which the right LED 303 is on, illuminatingthrough the tooth for detection/capture (180°) by the camera 305 on theleft. FIG. 3D is similar to FIG. 3A, showing light applied from theright side passing into the tooth (arrows) and either passing through tothe camera sensor 305 (also referred to herein as an image sensor,camera, or just “sensor”), or scattered from an internal region. Theorientation of the camera sensor and illumination source may beswitched. In FIG. 3B the left LED 303′ is on, illuminating through thetooth for detection/capture (180°) by the camera 305′ on the right. InFIG. 3C, both of the LEDs 303, 303′ are on, illuminating from both rightand left sides, and a camera 305″ located 90° off of the axis of theLEDs captures the trans-illumination image.

In general, the trans-illumination imaging data such as that describedabove can be combined with, and collected concurrently with, 3D surfacedata (e.g., 3D surface model data) of the teeth, allowing an additionallayer of data on internal structures such as caries and cracks. Further,the use of multiple projections (taken from multiple orientations) asdescribed may enable reconstruction of volumetric models internalstructures of the teeth enamel, showing features that would nototherwise be visible.

Although the 90° and 180° configurations of trans-illumination of theteeth may be useful, it may be particularly beneficial to providepenetration imaging configurations in which the angle between theemitted and received rays (vectors) is much smaller, e.g., between 0°and 30°, between 0° and 25°, between 0° and 20°, between 0° and 15°,between 0° and 10°, etc. In particular, angles between 0° and 15° (orbetween >0° and 15°) may be useful.

Trans-illumination in the 180° configuration and 90° configuration mayconstrain the movement of the intraoral scanner wand around the teethdue to their camera to light source angle constraint (as shown in FIGS.2C and 2D). Thus, also described herein are methods and apparatuses forpenetration imaging/visualization, e.g., of the enamel-dentin area,using a small angle, including between 0° and 15°. In one example, alight source (LED) emitting a penetrative spectral range (e.g., 850 nm)is used having a viewing vector at a small angle of 0°-15° relative tothe camera view angle. As mentioned, this penetration imaging may becombined with concurrent 3D surface modeling of the teeth. The relativepositions of the light source(s) and cameras(s) are typically known, andone or more penetration images may be taken at each position of thewand. Because of the small angle of the viewing vectors that may be usedby the wand, the intraoral scanning wand may be configured with just aslight curve, allowing it to fit and be easily maneuvered around theintraoral cavity, unlike wands configured to measure 90° and 180°trans-illumination, which may use a device geometry including side wingsto hold the LEDs and sensor(s) so that the wand can wrap around thetooth for the imaging (e.g., see FIG. 2C). The use of small-anglereflectance imaging may enable scanning in buccal and lingualdirections, whereas the 90 degree (trans-illumination) scanning asdescribed herein may be limited to scanning in the occlusal direction.

The use of a small angle for penetration imaging may include imaginginto the tooth using the wand in a way that enables unconstraintmovement around the tooth, and may enable capturing the internalstructure data while also scanning for 3D (surface) model data, withoutrequiring a dedicated structure and/or mode of operation. However, theuse of small angles between the emitting light and the detector(s) mayalso be complicated by direct reflections. For example, directreflection may occur in regions on the surface of the tooth in which theangle between the illumination and the imaging angles are approximatelyequal (e.g., in the cone of light and imaging NA). These directreflections may be problematic if they saturate the sensor, or if theyshow surface information but obscure deeper structure information. Toovercome these problems, the apparatus and methods of using themdescribed herein may capture and use multiple illumination orientationstaken from the same position. As used herein, in the context of ahand-held wand, taking multiple images from the same position mayeffectively mean taking multiple images at approximately the same time,so that a significant amount of movement has not occurred. For example,the images may be taken within a few milliseconds (less than 500 msec,less than 400 msec, less than 300 msec, less than 200 msec, less than100 msec, less than 50 msec, etc.) of each other, and/or correcting forsmall movements.

Alternatively or additionally, the apparatuses and/or methods may reduceor eliminate the problems arising from saturation with direct reflectionby using only the non-saturated pixels. In some variations, the surfaceinformation may be subtracted from the penetration images as part of theprocess. For example, visible light images (“viewfinder images”) orsurface imaging may be used to remove direct surface reflections.

In general, the apparatuses (e.g., systems) described herein may knowthe position of the wand at all times based on the surface scan, evenwhen taking images at different (even small angle) angles. Thus, whenperforming surface and penetrating scans concurrently or nearlyconcurrently (e.g., within 600 ms, 500 ms, 400 ms, etc. of each other),including interleaving these scans with other scanning types, theposition of the wand may be known relative to the object(s) beingscanned. Based on this information, the apparatus may estimate whichpart(s) of the multiple images or signals is/are arriving from thesurface and what is/are arriving from deeper structures.

FIG. 4A illustrates an example of a configuration of penetrative lightsources 403, 403′(e.g., penetrative spectral range light sources) andcamera(s) 405 that may be used as part of an intraoral scanner wand,shown in different positions around the target object (tooth 401). InFIG. 4A, three camera positions are shown, and each in each position thecamera is flanked by the pair of LEDs (e.g., 403 and 403′) for emittinglight in the penetrative spectral range (penetrative wavelength).Alternatively a single light source (e.g., LED) may be used instead of apair. Different images using the penetrative modality may be taken atdifferent wand positions relative to the teeth. Alternatively, the wandmay be configured with multiple imaging sensors (cameras) and multiplelight sources, allowing multiple penetration images may be taken atapproximately the same time, e.g., by turning on multiple sensors whenilluminating from one or more LED orientations (e.g., FIGS. 5G and 5E,etc.). In FIGS. 5A-5I, at least nine different orientations ofpenetration images may be taken, as shown. Alternatively oradditionally, multiple orientations may be taken sequentially, includingwithin a very short time period (e.g., within <500 ms, 400 ms, <300 ms,etc.).

FIGS. 4B-4F illustrate other emitters and detectors for use with of anyof the penetrating wavelengths that may be used to take images into theobject having semi-transparent strongly scattering regions (e.g.,teeth). These images typically collect reflective mode (e.g., light at apenetrative wavelength that has passed into the tooth, and beenscattered/reflected from internal structures so that it can be collectedby the detector. In FIG. 4B a combination of classic (e.g., 90°, 180°)trans-illumination and small-angle illumination angles are included. InFIGS. 4C-4F the angle of the ray of light emitted and collected is verysmall (e.g., around 0°) and can be collected by placing the emitter 403,403′ and detector 405 assembly (e.g., CMOS, CCD, etc.) adjacent to eachother, as shown in FIG. 4C, combined with each other, as shown in FIG.4D, or simply sharing a common or near-common beam path, as shown inFIGS. 4E and 4F, which may use reflection or waveguides to directemitted and/or received light, including the use of beam splitters(dichroic beam splitters) and/or filters.

As mentioned above, any appropriate sensor may be used, including CMOSor CCD cameras, or any other sensor that is capable of detecting theappropriate wavelength, such as near-IR wavelength detectors.

Although applying a penetrative illumination from nearby the sensor(camera) may result in the strongest illumination in the region nearestto the camera, and therefore an unequal distribution of illumination,this is surprisingly less problematic then was expected. In penetrationimaging conditions, the light generating the captured image has traveledthough the object, and the longer the path, the longer the scatteringthat will occur, resulting in a more smoothed-out illumination whencompared to direct illumination. In front illumination, as results withsmall-angle illumination, the strongest amount of light will be presentin the region nearest to the illuminator (e.g., LED), which will backscatter; this nearby region (e.g., the first 1-2 mm) is an importantregion for detecting caries. However, it may still be desirable tocompensate for the resulting non-uniform illumination profiledistribution, as discussed above.

The use of penetration imaging, and particularly small angleillumination/imaging, which may also be described as reflective imaging,may provide information about internal regions (such as cracks, caries,lesions, etc.) of the teeth that would not otherwise be available. Theinternal feature (or internal region) information may be incorporatedinto a 3D model, which may be particularly powerful when combined withsurface information (e.g., the 3D surface model or depth information).This may allow the user to capture the diagnostics data seamlesslyduring the 3D scanning procedure while allowing unconstrained movementaround the teeth to capture data from different angles, providing a 3Dmodel of the tooth interior.

Combining Surface Data with Internal Feature Data

As mentioned above, it may be particularly beneficial to combine and/orcoordinate 3D surface data with any of the internal feature data(including, but not limited to, penetration imaging data). For example,internal feature data such as penetration imaging data may be combinedwith surface data (surface imaging data) collected from the same orapproximately the same position of an intraoral scanner so that the samecoordinate system may be applied to both types of data.

As described above, a color 3D intraoral scanner such as the one shownin FIG. 1A, may be equipped with illumination devices emitting light attwo or more different spectral ranges for capturing a variety of surfaceand internal features. The data (e.g., surface data and internal featuredata) collected may be correlated and combined to form a 3D modelincluding information about lesions, decay, and enamel infractions aswell as teeth internal structure. The internal feature data may begathered by any appropriate penetrative imaging technique, including thereflective (e.g., small-angle) illumination and imaging, andtrans-illumination imaging techniques described above or by othertechniques known in the art, including, but not limited to UV/bluefluorescence and red light fluorescence.

The internal feature data may be collected (and may include lesion andinternal teeth structure images) and combined with the surface dataincluding color 3D surface model data for the teeth. The combination ofsurface and internal data may be expressed as a 3D model or 3Drendering, which may include a full color 3D data (including models andrenderings) of the lesions and tooth internal structure as well as thesurface of the teeth, gums and any other scanned portion of theintraoral region. Although in some variations the internal and surfacedata may be coextensive, in some variations the surface data may be moreextensive than the internal data; for example, the 3D model may includeinternal data for only a portion of the 3D model, while other regionsmay not include (or may include only incomplete) internal features.

In use, a 3D model of a tooth or teeth including both surface andinternal elements may be analyzed either automatically or manually, andinternal features may be identified and/or marked. For example, lesions,caries and/or cracks may be labeled, including color coding, e.g.,according to their type and level of risk they represent in one or moreimages that may be provided and/or as part of a data file that isgenerate to show these images. Alternatively or additionally, a writtentranscript/description of these findings may be provided.

An intraoral scanner for generating a 3D model including both surfaceand internal structure as described herein may include one or more imagesensors. For example, the image sensor may be configured for capturingcolor 3D (surface) images or data, and may also capture lesion and teethinternal structure images. Optionally or additionally, the system mayhave multiple sensors. The surface data may be acquired using anintraoral scanner in any appropriate manner. The intraoral scanner isgenerally configured to scan (via the wand) in both surface and internalimaging modes, including scanning concurrently. For example, surfacedata may be captured using a color intraoral 3D scanner by confocal,stereo vision or structured light triangulation or any other 3D surfacescanning technology capable of intraoral scanning.

As illustrated in FIGS. 10A and 10B, the illumination light sources(including the lights sources for the first modality (e.g., surfacescanning), for the second modality (e.g., penetrative imaging such aspenetration imaging), and/or for the third modality (e.g., colorscanning) may be located at the front tip of the intraoral scanner wand,e.g., near the scanned objects or inside the scanner head. The front tipillumination configuration may be configurable according to theapplication needs with or without any particular light source suitablefor the desired diagnostics feature by changing the front tip. The lightsource(s) and the sensors (e.g., cameras) may be arranged in anyappropriate manner, including as shown in FIGS. 10A-10B and 4. Forexample, the light sources and cameras may be adjacent to each other. Insome variations the system or method uses miniature sensors 1005, 1007,e.g., located at the front tip in a wrap-around manner, to capturestereoscopic 3D internal feature data (e.g., images) and/or forfacilitating penetration imaging in a more efficient fashion.

As mentioned, in some variations, the lesion/internal tooth structurecapture methods may be any combination through-tooth penetrationimaging, including one or more of: trans-illumination, red light laserfluorescence and blue/UV laser fluorescence, etc. In general, theinternal feature data may be used in combination with the surface data,including the coordinate system of the surface data, to reconstruct a 3Drepresentation of the tooth structure. For example a 3D reconstructionof the tooth data may be reconstructed by an algorithm combining several(e.g., multiple) 2D images using the any of the internal feature imagingtechniques described herein, typically taken at several different anglesor orientations.

Data captured by the intraoral scanner, including in particular the 3Dmodel of the tooth/teeth having both surface and internal features, maybe stored by the device and/or transmitted to a physician, medicalrecord, dentist, or the like. For example, any of the data captured bythe intraoral scanner, i.e. a color 3D model combining the topography ofthe teeth lesions and internal teeth structure, may be maintained in adesignated patient database for longitudinal monitoring and preservationof patient's oral health. The data may be annotated (including datingand/or markings referencing internal features) or unannotated.

For example, longitudinal comparison in time may be done using the 3Dmodels described herein at one or more levels, including by comparingacross time: surface changes, visual color changes, internal/volumetricchanges, or any combination of these. For example, each can be shown asbefore and after e.g., by manual evaluation, or subtracted and comparedautomatically. In some embodiments, two or more 3D models may besuperimposed with one another on a display to highlight differencesbetween the 3D models. The superimposed models may help highlightchanges in enamel thickness, dentin volume, color, opacity, and/ordecreases/increases in caries size, for example. Optionally, a 3D modelof a patient's dentition from an earlier date may be morphed into a 3Dmodel of the patient's dentition at a later date to help highlight anychanges in the patient's dentition over time. In some embodiments, atime series of 3D models may be progressively morphed from one to thenext to provide a video or animation of the changes in the patient'sdentition. Automatic comparison may be done by applying or converting toa common coordinate system, which may in particular be done usingsurface information (e.g., based on the 3D surface model data that isincluded as part of the generated 3D volumetric model). Typically, allthree types of data (surface, color, volumetric, etc.) areinterconnected by the same coordinate system, as already describedabove. In general the method and apparatuses described herein, includingthe 3D models, may be used to predict future dental or orthodonticconditions in a patient as described, for example, in U.S. 2016/0135925,incorporated by reference in its entirety.

When comparing scans, including 3D volumetric scans, the scans may beadjusted or normalized relative to each other for automatic,semi-automatic or manual comparison. For example, a scan of the tooth orteeth (e.g., a full jaw scan, partial scan, etc.), may not be 100%repeatable, particularly to a precision higher than the voxelresolution. To compare voxel-by-voxel, a matching and/or morphingfunction may be applied to one or both scans to allow more directcomparison. For example, a matching and/or morphing function may beused. A morphing function may bring the external surfaces to match andalign, allowing a voxel-to-voxel comparison. This may also allowcomparison of full scans to partial scans.

As mentioned above, in general, captured data may be stored and saved inthe same coordinate system. Thus, surface data (including 3D surfacemodel data) may use a coordinate system (e.g., x, y, z; so that the 3Dsurface model is S(x,y,z)) and the internal feature data may use orreference the same coordinate system (e.g., so that the internal featuredata is I(x, y, z)). Thus, common features or structures may have thesame address (coordinates) between both data sets.

FIG. 6 is a diagram illustrating an example of a method for generating a3D model or rendering of a tooth or teeth using surface data andinternal feature data. In this example, a hand-held intraoral scanningwand (scanner) may first be positioned adjacent to a target intraoralregion 601 to being scanning. Once scanning is initiated, the apparatusmay collect surface data (e.g., 3D model surface data) including depthinformation in a first coordinate system 603. The surface data maytypically be collected while illuminating the sample using a firstillumination spectrum, such as visible light (e.g., monochromatic orbroadband light). Internal feature data may also be collected, e.g.,using a second illumination spectrum (which may include just a singlewavelength or small range of wavelengths) that is/are penetrative intothe tooth/teeth 605. This data may use the same coordinate system as thesurface data, which may be accomplished as described in greater detailbelow. Once collected, the data may be analyzed, and/or filtered(including subtracting, smoothing, etc.), and combined to form a 3Dmodel rendering of the intraoral cavity (e.g., tooth, teeth, gums, jaw,etc.) using both the surface data and the internal feature data 607. Forexample, when building the 3D geometry of the internal feature data(which is typically two-dimensional in nature), the algorithm may usethe reference to the known 3D surface scan to improve the accuracy ofthe internal feature data.

In general, in any of the apparatuses and methods described herein, theinternal feature data collected 605 may be used to reconstruct avolumetric model of the tooth or teeth including the internal features.In particular, tomographic reconstruction (e.g., optical tomography) maybe used. A fully volumetric modeling may be used. Typically, everypenetrating light ray can either be refracted, reflected, scatteredand/or absorbed (including combinations of these), depending on thematerial properties and the light used. In some variation, the methodsand/or apparatus may divide the volume of the tooth into small voxelsand for each voxel, estimate these four parameters (refraction index,reflection, scattering, absorption) based on the imaging data collected,using the coordinate system corresponding to the coordinate system ofthe surface data. More complex models (e.g., based on non-isotropicscattering or complex surface scattering) may also be used. Once a setof parameters for each voxel is estimated, the method or apparatus maycompare how well the captured images, fit this model. Thus in somevariations the apparatus and/or method may seek to minimize thedifference between the captured images and the modeled, predicted image.An initial guess may be built from the 3D surface capture, includingestimates of enamel parameters and width.

Alternatively or additionally, multi-surface modeling may be used.Multi-surface modeling assumes a set of material (and in some casesuniform) in optical properties, such as properties for air, dentin, andenamel (but may include more than these three). This technique may seekto find the boundaries between the materials. There are multiple ways toaccomplish this, including using techniques similar to what is describedabove for the full volumetric modeling, but without the voxelsrepresentation. Alternatively or additionally, a contour line method maybe used in which a first (e.g., air-enamel) boundary is given from the3D surface capture, and then, by finding the edges of regions in the 2Dpenetrating images, a smooth 3D surface may be approximated that bestfits this silhouette. See for example “3D Shape from Silhouette Pointsin Registered 2D Images Using Conjugate Gradient Method. AndrzejSzymczaka, William Hoffb and Mohamed Mahfouzc,” the entire contents ofwhich are incorporated herein by reference. Apart from contours, otherfeatures, like points, corners, as known in the art, may be used. Thesefeatures may be detected from the different viewpoints, and located in3D by triangulation, and are part of the boundaries.

In practice, recording the surface data and internal feature data in thesame coordinate system may be achieved by scanning both the surface andthe internal features at the same position and/or time. As mentioned, ina hand-held user controlled intraoral scanning device (e.g., wand) itmay be difficult to scan the same region at different times in differentwavelengths. Thus, any of the apparatuses and methods described hereinmay coordinate scanning at the different modalities or modes (e.g.,surface data scanning and/or internal features/penetrative datascanning).

For example, FIG. 7 illustrates one method in which the intraoralscanner alternates between surface scanning and one or more otherscanning modalities (e.g., internal feature scanning, such aspenetration imaging scanning). In FIG. 7, after positioning the scanneradjacent to the target intraoral structure to be modeled 701, the wandmay be moved over the target while the apparatus automatically scans 703the target for both surface data and internal data. As part of thismethod, the system may alternate (switch) between scanning a portion ofthe tooth using a first modality 705 (e.g., surface scanning, usingemitting light in an appropriate wavelength of range of wavelengths) tocollect surface data such as 3D surface model data and scanning with asecond modality 707 (e.g. a penetrative wavelength). After anappropriate duration in the first modality, the method and apparatus maybriefly switch to a second modality (e.g., a penetrative wavelength orrange of wavelengths) to collect internal feature data for a brief timeperiod (second duration) 707 over approximately the same region of theobject scanned in the surface mode. At the time of the switch, thecoordinate system between the two modalities is approximately the sameand the wand is in approximately the same position, as long as thesecond duration is appropriately short (e.g., less than 500 msec, lessthan 400 msec, less than 300 msec, etc., less than 200 msec, less than100 msec, less than 50 msec, etc.). Alternatively or additionally, themethod and apparatus may extrapolate the position of the wand relativeto the surface, based on the surface data information collectedimmediately before and after collecting the internal data. Thus, in anyof the methods described herein, including as shown in step 703 of FIG.7, the apparatus may interpolate the positions between each scan (e.g.,first modality scan, such as a surface scan, a second modality scan,such as a penetrative, e.g., near-IR scan or scan(s) and a thirdmodality scan, such as a color scan, etc.). This interpolation maycorrect for the small but potentially significant movement of the wandduring scanning. In particular, when coordinating between the surfaceand internal structures, in which the scanning is being manuallyperformed, interpolating (and/or extrapolating) to approximate the moreaccurate 3D position of the teeth (or of the teeth relative to thescanning wand) for each scanned image. The portion of the teeth scannedusing a penetrative wavelength may therefore be interpolatedproportionally between the surface scans done before and after thepenetrative scan(s). See, e.g., FIG. 8, described below, showing anexemplary relative timing of the scans in each mode. Alternatively oradditionally, the position of the teeth and/or wand/scanner during ascan may be extrapolated from the prior surface scan position based onthe rate of movement of the scanning wand (e.g., as estimated from therate of change across the surface from prior surface scans, and/ormotion sensor(s) in the wand). Correcting the coordinate system of eachscan in this manner (for example, in x, y and z position, andorientations angles) may allow images in different modalities to betightly registered relative to each other, regardless of how the scanneris manipulated by the user. In penetrative scans, in which multiplescans may be taken from the same relative position and used toreconstruct internal features, the accuracy of the coordinate system mayallow higher resolution modeling of the internal features.

In general, when collecting penetrative wavelength images, the lightemitted and received may have different polarizations. In the reflectivelight mode, for example when using small-angle penetration imaging, someof the energy is penetrating, but some is also reflected from thesurface. It may be preferable to block this direct surface reflection,which may be done in any appropriate manner, including usingpolarization. For example, to block the surface reflection the sample(e.g., tooth) may be illuminated with a penetrative wavelength at aspecific polarization, and this polarization may be blocked in theimaging path. This polarization may also be helpful to block directlight from the illumination source in trans-illumination (e.g., wherethere is a direct line of sight to the illuminator as in 180°trans-illumination).

Although many of the methods and apparatuses described herein includeswitching between modes to distinguish surface and internal structures,in some variations, they may be truly simultaneously detected, forexample, using a dichroic beam splitter and/or filter. Thus, byseparating out the wavelengths and/or polarization that are penetrativeand include internal reflections and/or scattering from those includingonly (or primarily) surface features, the surface data may be collectedand processed separately from the internal features, and these two datasets may be recombined later; this technique may inherently use the samecoordinate system.

For example, FIG. 2E shows a schematic of intraoral scanner configuredto do both surface scanning (e.g., visible light, non-penetrative) andpenetrative scanning using a near infra-red (NIR) wavelength (at 850 nmin this example). In FIG. 2E, the scanner includes a near-IRillumination light 289 and a first polarizer 281 and a second polarizer283 in front of the image sensor 285 to block near-IR light reflectedoff the surface of the tooth 290 (P-polarization light) while stillcollecting near-IR light scattered from internal toothstructures/regions (S-polarization light). The NIR light illuminates thetooth in P-polarization, and specular light reflected from the surfaceof the tooth, e.g., the enamel, is reflected with specular reflectionhence its P-polarization state is conserved. Near-IR light penetratingthe internal tooth features, such as the dentin, is scattered resultingin random polarization (S and P). The wavelength selective quarterwaveplate 293 does not modify the polarization of the near-IR light(e.g., it leaves the polarization state of the near-IR light beingdelivered unchanged) but changes the polarization of the returning scanlight from P to S such that only surface reflection are captured in thescan wavelength. The returning near-IR light, having a mixture of S andP polarizations, is first filtered through the polarization beamsplitter (PBS) 294 and polarizing filter 283 such that only theS-polarization is transmitted to the image sensor. Thus only the near-IRS-polarization light, coming from the tooth internal structures, iscaptured by the image sensor while specular light, having the originalp-polarization, is blocked. Other intraoral scanner configurations withor without polarization filters such as those shown in FIG. 2E may beused as part of the probe.

In FIG. 2E, the surface scan may be performed by illuminating thesurface (using the scanner illumination unit 297), illuminating inp-polarization, and the polarization is reversed by thewavelength-selective quarter waveplate 293 (transmitting S-polarizationlight to the image sensor).

As shown in FIG. 7, the scanning scheme, including the duration of thescanning modalities such as the second scanning modality to determineinternal feature data, may be manually or automatically adjusted 709.For example, scanning procedure (time sharing and sequence) may bevaried per case and the system may automatically optimize the scanningresources so as to get high-quality scans and/or more completereconstructions. The method or apparatus may determine the quality ofthe scanned data 709, such as the quality of the scanned surface data,and may adjust the scanning duration(s) (e.g., the second duration)accordingly. An estimate of quality may be made automatically, forexample, based on blurring, over- or under-saturation, etc. For example,the duration of a scanning scheme may be dynamically adjusted (e.g.,increased or decreased) based on the quality of the scans in thismodality; if the prior x scans in this modality are below a first (e.g.,minimum) quality threshold (quantifying one or more of: blurring,over-saturation, under-saturation, etc.) the scan duration for thatmodality, d_(i), may be increased. Scan time may be reduced if theduration of the scan is above a minimum duration and the quality isabove a second quality threshold (which may be the same as the firstquality threshold or greater than the first quality threshold). Reducingthe scan duration may allow the duration of other scanning modalities toincrease and/or the rate of switching between scanning modalities toincrease. Alternatively or additionally, the scan duration for amodality may be adjusted based on the completeness of the 3D model beingreconstructed. For example, when scanning a region of the 3D modelhaving a more complete surface model (e.g., regions over which thesurface model has already been made), the duration of the surface scanmay be decreased, and the duration of the penetrative scan (e.g., areflective scan using a near-IR wavelength, or a trans-illumination scanusing a near-IR wavelength) may be increased to increase the resolutionand/or extent of the internal structures. Similarly, the frequency ofthe scanning in each mode may be adjusted dynamically by the apparatus.Any of the methods and apparatuses described herein may also beconfigured to give feedback to the user to slow down or add scans from aspecific angle by showing these missing regions or angles in the 3Dgraphical display.

As illustrated in FIG. 7 (e.g., optional step 708) and in FIG. 8, morethan two scanning modalities may be used. FIG. 8 illustrates anexemplary method of operating an intraoral scanner so that it switchesbetween different scanning modalities, including surface scanning 801,laser florescence 803, color visible light scan (viewfinder) 805,penetration scanning 807, UV scanning, etc. The system may initiallyswitch between the scanning modalities with a default scanning scheme;as mentioned, the system may then (in real time) analyze the data comingfrom each of the scanning modalities and may prioritize the scanningmodalities that have less complete data, for example, by expanding thefrequency and/or duration (d) that they are scanned. In someembodiments, the system may compare the gathered data from the one ormore of the scanning modalities to predetermined data resolutionthresholds in order to determine which scanning modalities toprioritize. For example, a system may increase the frequency or durationof surface penetrative imaging after determining that sufficient surfacedata had been gathered with a surface imaging modality and that internalfeature data resolution is still insufficient. Alternatively oradditionally, in some variations scanning may be done for differentmodalities simultaneously. Once sufficient scanning area has beencompleted, the combined 3D model of the intraoral region may beassembled using the scanned data 711; alternatively the 3D model may becontinuously assembled as the scanning is ongoing. The frequency of thescanning 809 is shown by the frequency of the scan amplitude in FIG. 8;surface scans are performed at the maximum of the scan amplitude andpenetrative scans at the minimum of the scan amplitude as the depth ofthe confocal scanning increases and decreases. The frequency of thedepth scanning 809 may be increased or decreased dynamically duringscanning. For example, to allow longer scanning duration scans, or toaccommodate for a faster-moving wand/scanner by the user. In somevariations the wand may include a motion sensor (e.g., an accelerometer,etc.) to detect movement rates, and the scanning rate(s) and duration(s)may be adjusted based on the detected motion of the scanner.

As shown in FIG. 6, the resulting 3D model including surface andinternal structures may be used in a variety of ways to benefit subject(e.g., patient) health care. For example, the 3D model may be used toidentify (automatically or manually) and analyze lesions, caries and/orcracks in the teeth. The 3D model may be used, for example, to measuresize shape and location of lesion including decay, to assess the type ofdecay based on translucently, color, shape, and/or to assess the type ofsurface issues based on surface illumination e.g. cracks, decay, etc.609.

This 3D data (or data derived from it) may be monitored over time for aparticular patient 611. For example, the 3D data may be checked forchanges in shape size and type over time either visually or using analgorithm.

In general, the 3D data may be annotated. For example, after a firstscan, a clinician may mark areas of interest which may be manually orautomatically assessed in following scans. In addition the 3D data maybe used to help treat or provide treatment guidance and monitoring 613.For example, if a clinician decides to restore a tooth, the 3D datashowing surface and internal regions generated as described herein maybe used to provide reduction guidelines for the tooth to ensure theremoval of the decayed volume. During the procedure, additional (e.g.,intermediate) scans may be made to provide the doctor with furtherdirection and immediate feedback on the reduction.

FIGS. 9A and 9B illustrate one example of a 3D model 900 rendering of anintraoral region of a subject including both surface (total surface isshown in the projection of FIG. 9A) and internal structures, shown inthe enlarged region in FIG. 9B. In FIG. 9B, the darker region 903apparent from the penetration imaging using 850 nm light combined withthe 3D surface data, shows a region of interest. The region of interestmay be a carious region or a dental filing, or the like. The ability tomanipulate images like this to rotate, zoom, section and otherwise viewthe 3D model or regions of the 3D model may greatly enhance thetreatment and understanding of a subject's dental needs.

Depth Scanning

FIGS. 11A-11I illustrates one example of volumetric modeling of internaltooth structure using a penetrative wavelength such as near-IRtrans-illumination (“TI”). In this example, a lesion in the tooth may bedetected when light is bellow lesion or at the level of the lesion. Whenthe light is below the lesion, the lesion absorbs the light, thus lesionappears as dark spot in the image. In FIG. 11D, a tooth having a lesionis shown with a scanner sensor 1101 above the tooth (positioned abovethe occlusive surface of the tooth). The scanner includes one or (asillustrated in FIGS. 11D-11F) two light sources (emitters) 1105, 1105′,emitting near-IR light, as shown by the arrows. The light penetrates thetooth and the sensor 1101 detects the occlusion of light due to thelesion, as shown in FIG. 11A.

Moving the scanner with the light source upwards (i.e., moving the wandof the scanner higher along the tooth) will produce a change in thelesion image as shown in FIG. 11B. The corresponding position of thelight sources relative to the tooth is shown in FIG. 11E schematically,and in the illustration of FIG. 11H. As the scanner is moved further upthe tooth, the dark spot representing the lesion 1113 will startshrinking until completely disappearing, turning into light saturation.Finally, when the light source 1105, 1105′ is above the lesion, the darkspot is no longer present (e.g., FIG. 11C) and only the centralocclusive region (the dentin) is shown. As already discussed above, theouter surface of the tooth and gingiva may be concurrently scanned usinga separate light source, providing the 3D outer surface of the tooth,and therefore the distance from the tooth to the scanner. Thisinformation, as described above, may be used to map the lesion's depthand/or shape.

Such depth scanning may be manually or automatically performed, and maybe useful for providing a backup and/or alternative to volumetricmodeling (e.g., 0-degree volumetric modeling) of the tooth/teeth. Indeedthis vertical scanning of the teeth (which may be performed in anydirection (bottom to top of tooth, top to bottom, etc.) may be used asone type or sub-type of volumetric scanning that may provide informationon shape and position of dentin and/or lesions.

For example, the method of vertically (z-axis) scanning of theteeth/tooth with an intraoral scanner, particularly one having both apenetrative (e.g., near-IR) and surface scanning wavelength(s), mayprovide an alternative method of volumetric scanning. In general, datamay be acquired by scanning up or down (in the z-axis) the tooth/teeth.

As discussed above, one configuration for the scanning devices describedmay optically image the inside region of a tooth/teeth using, e.g.,trans-illumination (through the sides) at an angle, such as a 90° angle,between light source and camera. When a dental caries is present in thetooth, viewing the tooth with a penetrative wavelength, e.g., intrans-illumination, from above (occlusion view) may reveal the caries asan occlusive region. Depending on the relative z (depth) position of thelight source with respect to the caries, an occluded regioncorresponding to the caries will be present in the x,y image. Thusscanning through the z-axis (depth) as described above may be used todetermine one or both of z-position and shape of the caries. In somevariations, a method for scanning using a penetrative wavelength (or apenetrative and surface scanning) may begin with illuminating from thesides and imaging from above and placing light as close as possible togum line. The method may then proceed to move up along the z axis oftooth, moving away from the tooth's occlusive surface. This may allowthe light to hit a lesion from different depths (in the z-axis). Asillustrated in FIGS. 11A-11C, a caries will be initially present, and asthe scanner is drawn upwards, may shrink in the imaging plane (x,y)until it is no longer blocking the light. Any of these methods may alsocalculate or determine the z-position along the tooth as the scanner ismoved upwards, so that the relative depth on the tooth is known, andtherefore the depth of the lesion is from the enamel layer. From thisinformation, the dimensions of the lesion may also be determined (e.g.,an estimate of how far along the z-position the lesion extends), as wellas the breadth and extent (e.g., how far it extends in x,y) may also bedetermined. Along with the surface 3D model, showing the outer shape ofthe tooth, this information may be used to provide a model of the toothand the overall lesion.

Thus, using both a penetrative wavelength (e.g., near IR) and thenon-penetrative (surface scanning) wavelength, a model of both theexternal and internal structures of the tooth may be determined. Depthscans (even non-contiguous scans) along the z-axis of the tooth may beparticularly useful for determining the depths and/or dimensions ofinternal structures within the tooth/teeth. In any of the methodsdescribed herein, as discussed above, a 3D scan of the tooth may beperformed concurrently with the penetrative (including depth) scanning.

Thus, in any of the methods of scanning a tooth as described herein, themethod may include determining a depth (z) dimension for each scan,showing the relative depth of the light source(s), e.g., the near-IRlight source(s) relative to the tooth. This information may be providedby the 3D surface scan corresponding/correlating to the penetrativescan. Depth information (e.g., knowing how much the scanner has beenmoved in the z-axis) may provide substantial volumetric information.

As mentioned above, the depth (z) scanning described herein may beperformed manually or automatically. For example, this scanning may beperformed by manually scanning the wand up and along the teeth. Duringscanning both concurrent 3D surface modeling and internalmodeling/imaging may be continuously performed during scanning. Anyappropriate scanning rate (e.g., 20 scans per second) may be done. Thus,a user may scan at a reasonable speed, and output may be done inreal-time, including displaying a lesion, and/or lesions (and any otherinternal structures) may be displayed later following analysis by thesoftware. In one example, concurrent scanning may be performed so thatthe surface scanning (using a laser) may be done for an approximately 35ms period, followed by a window of 15 ms for other types of imaging,including color, near IR, etc., and repeated during the scanning period.In some examples, the near-IR scanning may be done for 5 ms within the15 ms window. Shorter sampling may be beneficial (e.g., shorter than 20ms, shorter than 15 ms, shorter than 12 ms, shorter than 10 ms, shorterthan 7 ms, shorter than 5 ms, etc.), as it may reduce smearing of theimage. However, shorter scan times may require higher energy, e.g., morepower/current to the penetrative light source. Imaging data may becollected throughout. Alternatively, scanning may be done for longer orshorter periods of time (e.g., surface scanning, near IR scanning, colorscanning, etc.), and/or at the same time (e.g., laser surface scanningand near-IR concurrently, using different emitters/detectors, forexample). In this manner, e.g., concurrent or rapid alternating (within200 ms, within 150 ms, within 100 ms, within 50 ms, etc.) of surface andpenetrative scanning, or any other different types of scanning, maypermit coordination between the surface (e.g., 3D) molding and internalstructures as described above.

Imaging Internal Structures Using Scattering Coefficients

Also described herein are methods and apparatuses for generating imagesof internal structures from within a tooth or other semi-transparent,strongly scattering object) based on a plurality of penetrative images(also referred to herein as “penetrating images”) through the object inwhich the position of the camera (relative to the object) is provided.These methods and apparatuses may therefore generate images, includingthree-dimensional models, of internal structures without requiring amodel of the external surface.

For example, described herein are methods and apparatuses, includingcomputing device readable media, for reconstructing a volumetricstructure from an object including semi-transparent strongly scatteringregions, such as a tooth. More specifically, these apparatuses (e.g.,systems) and methods may provide techniques for reconstructing an innerstructure of an object, such as the dentin in the teeth.

Generally, objects that are semi-transparent and strongly scattering toa specific wavelength can be imaged according to the methods (and usingany of the apparatuses) described herein. If the location andorientation of the camera with respect to the object is known, the innerstructure of the object can be reconstructed with a low computationalcomplexity proportional to the volume being reconstructed and the numberof images.

Any of the intraoral scanners that take images through a subject'sintraoral region (e.g., tooth or teeth, gums, jaw, etc.) describedherein and also provide information on the relative position of thescanner (e.g., the camera of the scanner taking the image), may be used.For example, returning to FIGS. 1A and 1B, FIG. 1A illustrates oneexample of an intraoral scanner 101 that may be configured or adapted asdescribed herein to generate 3D models having both surface and internalfeatures. As shown schematically in FIG. 1B, an exemplary intraoralscanner may include a wand 103 that can be hand-held by an operator(e.g., dentist, dental hygienist, technician, etc.) and moved over asubject's tooth or teeth to scan both surface and internal structures.The wand may include one or more sensors 105 (e.g., cameras such asCMOS, CCDs, detectors, etc.) and one or more light sources 109, 110,111.

In FIG. 1B, two separate light sources are shown: a first light source109 configured to emit light in a first spectral range for detection ofsurface features (e.g., visible light, monochromatic visible light,etc.) and a second light source 111 configured to emit light in a secondspectral range for detection of internal features within the tooth(e.g., by trans-illumination, small-angle penetration imaging, laserflorescence, etc., which may generically be referred to as penetrationimaging). Although separate illumination sources are shown in FIG. 1B,in some variations a selectable light source may be used. The lightsource may be any appropriate light source, including LED, fiber optic,etc. The wand 103 may include one or more controls (buttons, switching,dials, touchscreens, etc.) to aid in control (e.g., turning the wandon/of, etc.); alternatively or additionally, one or more controls, notshown, may be present on other parts of the intraoral scanner, such as afoot petal, keyboard, console, touchscreen, etc.

In addition, the wand 103 may also include one or more position and/ororientation sensors 123, such as an accelerometer, magnetic fieldsensor, gyroscope sensors, GPS etc. Alternatively or additionally, thewand may include an optical sensor, magnetic sensor, or other somecombination thereof, for detecting the relative position of the wand,and particularly of the camera(s) with respect to the object beingimaged (e.g., a tooth or teeth). Alternatively or additionally, theapparatus may detect the relative position of the wand based on thesurface images (e.g., surface scanning) and/or viewfinding scan taken asdescribed above.

In general, any appropriate light source may be used, in particular,light sources matched to the mode being detected. For example, any ofthese apparatuses may include a visible light source or other lightsource for surface detection (e.g., at or around 680 nm or otherappropriate wavelengths), a visible light source (e.g., white lightsource of light) for traditional imaging, including color imaging,and/or a penetrating light source for penetration imaging (e.g.,infrared and/or near infrared light source).

The relative positions of the light source(s) and cameras(s) aretypically known, and one or more penetration images may be taken at eachposition of the wand. The positions of the light source(s) and camera(s)can include three numerical coordinates (e.g., x, y, z) in athree-dimensional space, and pitch, yaw, and roll of the camera.

The intraoral scanner 101 may also include one or more processors,including linked processors or remote processors, for both controllingthe wand 103 operation, including coordinating the scanning and inreviewing and processing the scanning and generation of the 3D modelincluding surface and internal features. As shown in FIG. 1B the one ormore processors 113 may include or may be coupled with a memory 115 forstoring scanned data (surface data, internal feature data, etc.).Communications circuitry 117, including wireless or wired communicationscircuitry may also be included for communicating with components of thesystem (including the wand) or external components, including externalprocessors. For example the system may be configured to send and receivescans or 3D models. One or more additional outputs 119 may also beincluded for outputting or presenting information, including displayscreens, printers, etc. As mentioned, inputs 121 (buttons, touchscreens,etc.) may be included and the apparatus may allow or request user inputfor controlling scanning and other operations.

Any of the apparatuses and methods described herein may be used to scanfor and identify internal structures such as cracks, caries (decay) andlesions in the enamel and/or dentin. Thus, any of the apparatusesdescribed herein may be configured to perform scans to detect internalstructures using a penetrative wavelength or spectral range ofpenetrative wavelengths. Although a variety of penetrative scanningtechniques (penetration imaging) may be used or incorporated into theapparatus, trans-illumination and small-angle penetration imaging, bothof which detect the passage of penetrative wavelengths of light throughthe tissue (e.g., through a tooth or teeth), may be of particularinterest.

The methods and apparatuses for visualization of the enamel-dentin areausing a penetrative wavelength (such as, for example, 850 nm) describedherein may acquire a plurality of projections or orientations from asingle position of the scanner relative to the tooth/teeth; inparticular three or more orientations or projections may be taken ateach position. Taking multiple (e.g., 3 or more) projections may providebetter imaging, as it may produce multiple (e.g., 3 or more) imagesthrough the tooth from a particular location of the wand relative to thetooth/teeth.

FIG. 12 illustrates an example of a portion of a scanner configured toinclude penetration light sources 1202, 1202′ (e.g., penetrativespectral range light) and cameras that may be used as part of anintraoral scanner wand. In FIG. 12, a camera 1200 is shown that isflanked by a pair of LEDs 1202, 1202′ for emitting light in thepenetrative spectral range in substantially the same direction as thecamera towards a target T (such as a tooth 1201). A single light source1202 (e.g., LED) may be used instead of a pair. In general according tothis disclosure, the light sources of the wand are projected insubstantially the same direction as the camera, but in some embodimentsthe light sources can vary +/−15 degrees from the direction of thecamera, as described above.

FIG. 13 shows a flowchart 1300 that describes one method forreconstructing a volumetric structure from an object havingsemi-transparent strongly scattering regions for a range of radiationwavelengths. The object having semi-transparent strongly scatteringregions can be, for example, a tooth comprising an exterior enamelsurface and an interior dentin surface.

At step 302 of flowchart 1300, the method comprises taking a pluralityof images of the object with a camera in the range of radiationwavelengths, wherein lighting for the plurality of images is projectedsubstantially from a direction of the camera. In some embodiments, therange of radiation wavelengths is an infrared or near infraredwavelength. The infrared or near infrared wavelength can be used, forexample, to penetrate the semi-transparent object. In one embodiment,the lighting for the plurality of images can vary +/−15 degrees from thedirection of the camera. The plurality of images can be stored incomputer memory coupled to the camera.

Any of these methods may also include receiving location datarepresenting a location of the camera relative to the object for each ofthe plurality of images. Generally, the location data includes theposition and orientation of the camera with respect to the object. Thislocation data can be determined from the plurality of images, oralternatively or additionally, the position and orientation can bemeasured with sensors 123 on the wand (e.g., gyroscope sensors,accelerometers, GPS, etc.). Alternatively or additionally, the positionand orientation can be computed by registration of scanned surface data.In some embodiments, the location data comprises three numericalcoordinates in a three-dimensional space (e.g., x, y, and z in aCartesian coordinate system), and pitch, yaw, and roll of the camera.The location data can also be quantified as vector metrics (e.g.,rotation metrics and vector position).

At step 306 of flowchart 1300, the method further comprises generatingfor each point in a volume an upper bound on a scattering coefficientfrom the plurality of images and the location data. Each of theplurality of images may be a projection from the real world (a 3Denvironment) onto a 2D plane (the image), during which process the depthis lost. Each 3D point corresponding to a specific image point may beconstrained to be on the line of sight of the camera. The real worldposition of each 3D point can be found as the intersection of two ormore projection rays through the process of triangulation.

In step 306, an upper bound on a scattering coefficient is determinedfor each point in a volume that represents the object being scanned. Theupper bound is selected from the plurality of images for each pointusing the location data from the camera to triangulate the position ofeach point. The plurality of images produces an intensity for each pointthat is a result of the amount of light reflected by the object. Thisintensity for each point is used to generate the scattering coefficientfor each point. The upper bound on the scattering coefficient for eachpoint can be stored in memory coupled to the camera.

Generating, for each point in the volume an upper bound on thescattering coefficients may include projecting each point of a 3D gridof points corresponding to the volume of the object onto each of theplurality images using a first calibration, producing a list ofscattering coefficient values for each projected point, correcting eachscattering coefficient value on the list of scattering coefficientvalues according to a volume response, and storing a minimum scatteringcoefficient value for each grid point from the list of scatteringcoefficient values.

A number of calibrations can be performed to facilitate projecting eachpoint of the 3D grid of points onto each of the plurality of images. Forexample, in one embodiment, the first calibration may comprise a fixedpattern noise calibration to calibrate for sensor issues and imageghosts of the camera. In another embodiment, the first calibrationcomprises a camera calibration that determines a transformation for thecamera that projects known points in space to points on an image. Insome embodiments, all of the calibrations described above can beperformed prior to projecting the points onto the images.

When generating an upper bound on a scattering coefficient from thepenetrative images and location data, the upper bound on the scatteringcoefficient(s) may only be determined for points within an exteriorsurface of the object being imaged. For example, the methods describedherein can further include receiving surface data representing anexterior surface of the object (e.g., scan data representing an exterioror enamel surface of a tooth). With the exterior surface data, onlypoints within this exterior surface (e.g., internal points) can be usedto generate scattering coefficients. This may allow the imaging to focusonly on, for example, a dentin surface within an enamel surface ofteeth.

Finally, any of these methods may comprise generating an image of theobject from the upper bound of scattering coefficients for each point308. Example of generating these images are provided herein, and mayinclude forming a line and/or surface based on threshold values of thescattering coefficients or values based on the scattering coefficients.

FIG. 14 is a flowchart 400 that illustrates a method for reconstructinga volumetric structure from a tooth. The tooth can be semi-transparentin a range of radiation wavelengths. At step 402, which is optional, themethod comprises receiving, in a processor, a representation of asurface of the tooth in a first coordinate system. The representation ofthe surface of the tooth can be, for example, a 3D model of the tooththat is produced either by scanning the teeth or by taking a mold of theteeth.

The method may also include receiving, in the processor, a plurality ofimages of the tooth in the range of radiation wavelengths, the pluralityof images taken with lighting projected substantially from a directionof a camera 404. In some embodiments, the wavelength is a penetrativewavelength of the infrared or near infrared region or a range within theIR/near IR. The infrared (IR) or near infrared wavelength can be used,for example, to penetrate the tooth. The lighting for the plurality ofimages can vary +/−15 degrees from the direction of the camera. Theplurality of images can be stored in computer memory coupled to thecamera.

At step 406 the method further comprises receiving, in the processor,location data representing a location of the camera for each of theplurality of images. Generally, the location data includes the positionand orientation of the camera with respect to the object. This locationdata can be determined from the plurality of images, or alternatively,the position and orientation can be measured with sensors on the camera(e.g., gyroscope sensors, accelerometers, GPS, etc.). Alternatively oradditionally, the position and orientation can be computed byregistration of scanned surface data. In some embodiments, the locationdata comprises three numerical coordinates in a three-dimensional space(e.g., x, y, and z in a Cartesian coordinate system), and pitch, yaw,and roll of the camera. The location data can also be quantified asvector metrics (e.g., rotation metrics and vector position).

The method may also include projecting each point of a grid of pointscorresponding to a volume within the surface of the tooth onto each ofthe plurality images using a first calibration 408. The grid of pointsthat is produced may be inside of the exterior surface of the tooth. Thegrid can sit on a cubic grid, for example. Each grid point can beprojected onto each of the plurality of images using a calibration. Anumber of calibrations can be performed to facilitate projecting eachpoint of the grid onto each of the plurality of images. For example, thecalibration may comprise a fixed pattern noise calibration to calibratefor sensor issues and image ghosts of the camera. In another embodiment,the calibration may comprise a camera calibration that determines atransformation for the camera that projects known points in space topoints on an image. In some embodiments, all of the calibrationsdescribed above can be performed prior to projecting the points onto theimages.

The method may further include producing a list of intensity values foreach projected point 410. The plurality of images produces an intensityfor each point that is a result of the amount of light reflected by theobject. This intensity value for each point may be stored.

At step 412 the method may further comprise converting each intensityvalue on the list of intensity values to a scattering coefficientaccording to a volume response. This step may be performed to calibratethe intensity value for each pixel. The process calculates a scatteringcoefficient that would produce such an intensity value for each pointrelative to the position of the camera. The output is a scatteringcoefficient which normalizes the intensity according to a volumeresponse.

Finally, in FIG. 14, the method may further include storing a minimumscattering coefficient for each point into a list of minimum scatteringcoefficients 414. The method may further comprise producing an imagefrom the list of minimum scattering coefficient for each point.

As described above, the methods and techniques can include a pluralityof calibrations to project points from the real world into the pluralityof images. One such calibration is an image fixed pattern noisecalibration (PRNU) which addresses sensor issues and system ghosts thatdo not depend on the object being scanned. FIGS. 15A-E show one exampleof an image fixed pattern noise calibration, which gives a constantresponse for a uniform plane target. FIG. 15A shows an original image ofa plane uniform target, including two particles 1501, 1502 in the middleof the image. FIG. 15B shows the median image after moving the targetparallel to the plane. This causes the two particles to “disappear” fromthe image. FIG. 15C shows the image after applying a bias coefficientfigure for each pixel, which creates strong electronic noise in theimage. In FIG. 15D, a slope has been applied to each pixel, resulting ina smooth pattern given by the optics. Finally, FIG. 15E shows the finalimage after response equalization.

Another calibration that may be applied is called a camera calibration,which allows the projection of real world (3D) points to 2D imagepixels. The camera calibration determines a transformation for thecamera that projects known points in space to points on an image.

A volumetric response calibration that gives a scattering coefficientfor all points in the world given an intensity in the image within afield of view of the camera may also be applied. This calibration bringsa standard scattering coefficient to constant response anywhere in thefield of view.

Finally, a scan to world camera calibration may be applied that is arigid body transformation that converts from the scan coordinate system(of the 3D scan of the object) to the camera calibration coordinatesystem (of the 2D images of the object).

Other techniques may be used to determine the volumetric scatteringcoefficients from the penetrative images and camera positions. Forexample in some variations, back propagation may be used. Backpropagation may include estimating (e.g., tracing) rays going throughthe tooth volume and entering the camera. The actual intensitiesreaching the sensor for each ray may be taken from the penetrativeimages and camera positions and orientations. For each ray the dampingof the intensity due to scattering in the volume it passes may beestimated. For example, the transmission of light through a stronglyscattering and weakly absorbing material may be modeled using a hybridcalculation scheme of scattering by the Monte Carlo method to obtain thetemporal variation of transmittance of the light through the material. Aset of projection data may be estimated by temporally extrapolating thedifference in the optical density between the absorbing object and anon-absorbing reference to the shortest time of flight. This techniquemay therefore give a difference in absorption coefficients. For example,see Yamada et al., “Simulation of fan-beam-type opticalcomputed-tomography imaging of strongly scattering and weakly absorbingmedia,” Appl. Opt. 32, 4808-4814 (1993). The volumetric scattering maythen be estimated by solving for the actual intensities reaching thesensor.

Any of the methods described herein may be performed by an apparatusincluding a data processing system (or subsystem), which may includehardware, software, and/or firmware for performing many of these stepsdescribed above, including as part of a processor of an intraoralscanner (see, e.g., FIG. 1B). For example, FIG. 16 is a simplified blockdiagram of a data processing sub-system 500. Data processing system 500typically includes at least one processor 502 which communicates with anumber of peripheral devices over bus subsystem 504. These peripheraldevices typically include a storage subsystem 506 (memory subsystem 508and file storage subsystem 514), a set of user interface input andoutput devices 518, and an interface to outside networks 516, includingthe public switched telephone network. This interface is shownschematically as “Modems and Network Interface” block 516, and iscoupled to corresponding interface devices in other data processingsystems over communication network interface 524. Data processing system500 may include a terminal or a low-end personal computer or a high-endpersonal computer, workstation or mainframe.

The user interface input devices may include a keyboard and may furtherinclude a pointing device and a scanner. The pointing device may be anindirect pointing device such as a mouse, trackball, touchpad, orgraphics tablet, or a direct pointing device such as a touchscreenincorporated into the display. Other types of user interface inputdevices, such as voice recognition systems, may be used.

User interface output devices may include a printer and a displaysubsystem, which includes a display controller and a display devicecoupled to the controller. The display device may be a cathode ray tube(CRT), a flat-panel device such as a liquid crystal display (LCD), or aprojection device. The display subsystem may also provide nonvisualdisplay such as audio output.

Storage subsystem 506 may maintain the basic programming and dataconstructs that provide the functionality of the present invention. Themethods described herein may be configured as software, firmware and/orhardware, and (of software/firmware) may be stored in storage subsystem506. Storage subsystem 506 typically comprises memory subsystem 508 andfile storage subsystem 514.

Memory subsystem 508 typically includes a number of memories including amain random access memory (RAM) 510 for storage of instructions and dataduring program execution and a read only memory (ROM) 512 in which fixedinstructions are stored. In the case of Macintosh-compatible personalcomputers the ROM would include portions of the operating system; in thecase of IBM-compatible personal computers, this would include the BIOS(basic input/output system).

File storage subsystem 514 may provide persistent (nonvolatile) storagefor program and data files, and may include at least one hard disk driveand at least one floppy disk drive (with associated removable media).There may also be other devices such as a CD-ROM drive and opticaldrives (all with their associated removable media). Additionally, thesystem may include drives of the type with removable media cartridges.One or more of the drives may be located at a remote location, such asin a server on a local area network or at a site on the Internet's WorldWide Web.

In this context, the term “bus subsystem” may be used generically so asto include any mechanism for letting the various components andsubsystems communicate with each other as intended. With the exceptionof the input devices and the display, the other components need not beat the same physical location. Thus, for example, portions of the filestorage system could be connected over various local-area or wide-areanetwork media, including telephone lines. Similarly, the input devicesand display need not be at the same location as the processor, althoughit is anticipated that the present invention will most often beimplemented in the context of PCS and workstations.

Bus subsystem 504 is shown schematically as a single bus, but a typicalsystem has a number of buses such as a local bus and one or moreexpansion buses (e.g., ADB, SCSI, ISA, EISA, MCA, NuBus, or PCI), aswell as serial and parallel ports. Network connections are usuallyestablished through a device such as a network adapter on one of theseexpansion buses or a modem on a serial port. The client computer may bea desktop system or a portable system.

Scanner 520 may correspond to the wand and other components responsiblefor scanning casts of the patient's teeth obtained either from thepatient or from an orthodontist and providing the scanned digital dataset information to data processing system 500 for further processing. Ina distributed environment, scanner 520 may be located at a remotelocation and communicate scanned digital data set information to dataprocessing system 500 over network interface 524.

Various alternatives, modifications, and equivalents may be used in lieuof the above components. Additionally, the techniques described here maybe implemented in hardware or software, or a combination of the two. Thetechniques may be implemented in computer programs executing onprogrammable computers that each includes a processor, a storage mediumreadable by the processor (including volatile and nonvolatile memoryand/or storage elements), and suitable input and output devices. Programcode is applied to data entered using an input device to perform thefunctions described and to generate output information. The outputinformation is applied to one or more output devices. Each program canbe implemented in a high level procedural or object-oriented programminglanguage to operate in conjunction with a computer system. However, theprograms can be implemented in assembly or machine language, if desired.In any case, the language may be a compiled or interpreted language.Each such computer program can be stored on a storage medium or device(e.g., CD-ROM, hard disk or magnetic diskette) that is readable by ageneral or special purpose programmable computer for configuring andoperating the computer when the storage medium or device is read by thecomputer to perform the procedures described. The system also may beimplemented as a computer-readable storage medium, configured with acomputer program, where the storage medium so configured causes acomputer to operate in a specific and predefined manner.

FIGS. 26A-26C and 27A-27G illustrate steps that may form part of amethod of forming a 3D volumetric model of a patient's teeth and may beused for one or more treatments using the methods and apparatusesdescribed above. In any of these methods, an intraoral scanner 2801capable of measuring both surface (including, in some variations color,e.g., R-G-B color) and internal structures may be used to scan thepatient's teeth (e.g., taking images and scans of the jaw, including theteeth). The apparatus may scan in different modalities, includingsurface (non-penetrative or not substantially penetrating, e.g., visiblelight, white light) and penetrative (e.g., near IR/IR) wavelengths.Scanning typically includes scanning from multiple positions around theoral cavity and assembling the resulting images into a three-dimensionalmodel of the teeth, e.g., by solving the relative position of the scansrelative to the jaw (FIG. 26C). The surface scanning may be used toconstruct a model (e.g., a 3D digital model, and/or renderings) of theouter surface of the jaw/teeth 2803, as shown in FIG. 26C.

In any of these methods and apparatuses described herein, internalstructures within the teeth may be formed or modeled to form avolumetric model of the teeth including the internal structures that areextracted from the penetrative scans (e.g., near-IR and/or IR scans), asillustrated in FIGS. 27A-27G. FIGS. 26A-27G describe one method ofreconstructing internal structures by using scattering coefficients(other methods may be used alternatively or additionally). In FIG. 27A,a grid is constructed of points representing the inner volume of thejaw/teeth. All of the grid points are projected onto the penetrative(e.g., near-IR) images taken, and all pixel positions may be saved foreach of the grid points, as shown in FIG. 27B. For each pixel positionand grid position, the apparatus may calculate the scatteringcoefficient which would result in the gray level of pixel observed, asgraphically illustrated in FIG. 27C. In the figures (e.g., FIG. 27C),the eye may represent the viewing angle of the sensor (e.g., camera).For each grid point, the apparatus may take the minimal scatteringcoefficient that is calculated (FIG. 27D). The grid of points withcorresponding minimal scattering coefficients may then provide a volume2909 that may be sampled at the grid points based on thresholds orcorrelations (e.g., iso-surfaces) of minimal scattering values, as shownin FIG. 27E. FIG. 27G shows an iso-surface 2911 created by identifying aconstant value of the density function sampled. FIG. 27F is an enlargedview of the same region of the teeth, showing both the iso-surface fromFIG. 27G as well as a ghosted image (partially transparent) of theenamel 2915 around the iso-surface. This iso-surface may representdentin and (as described below) dental caries extending from the outersurface of the tooth toward the dentin.

In the example shown in FIG. 27F, the iso-surface shows thedentin-enamel transition 2911 visible beneath the enamel 2915. Theexample in FIG. 27F also indicates a dental caries shown in circledregion 2913. In this example, the dental caries (similar to the dentin)appears as an iso-surface within or surrounded by the enamel. The dentalcaries may be distinguished because it extends from the inner, dentinregion to an outer surface of the tooth. Since the methods andapparatuses described herein may accurately reconstruct both the outersurface and the inner structures, this characteristic configuration(showing an arm or extension extending from the outer surface throughthe IR/near-IR transparent enamel) may be used to identify dentalcaries. In FIG. 27F a likely dental caries region is circled 2913,showing an extension or bridge between two teeth in a region where thesurface scan shows that the teeth are actually separate. Thus, combiningthe surface scan with the internal scanning (e.g., from the IR/near-IRimages) may allow for corrections in the internal data due to errorsthat may occur because of the limited view angles or the like. Any ofthe apparatuses and methods described herein may be configured toautomatically or semi-automatically identify these regions orirregularities corresponding to dental caries and the like. They may behighlighted in the model, image or representation of the teeth, and/or aflag, alert or other notification, along with a putative location, maybe presented, transmitted and/or stored. Alternatively or additionally,the threshold(s) used to determine the iso-surfaces may be chosen todistinguish between the one or more internal features such as thedentin, caries, fillings, cracks, etc.

Alternatively or additionally, the apparatus may automatically (orsemi-automatically) determine and distinguish internal structures withinthe teeth based on the shape of the iso-surfaces and/or their relativeposition(s) within the teeth. As mentioned above, caries may have asimilar densities (e.g., scattering coefficients) compared to dentin.However, the morphology of the caries may distinguish them from dentin.The apparatus may detect ‘arms’ or appendages of material having adensity (e.g., scattering coefficients) similar to that for dentin, butextending from the out surface of the enamel. Since the outer surface ofthe teeth may be well characterized in addition to the internalstructures, the extent of a caries may be determined by mapping theouter surface of the iso-density map for regions extending from theouter surface toward a larger, defined internal dentin pattern. Theborder between the dentin and the internal extent of the caries may bedetermined by approximating the continuous surface of the dentin,including the region around the “projecting” region and/or looking atthe rate of change of direction of the surface of the dentin. Otherinternal structures, such as fillings, cracks and the like may bedistinguished based on their scattering coefficient value ranges, and/orbased on their position or morphology. The apparatus may display them indifferent colors, annotations, etc.

Thus, in any of these methods and apparatuses, the scanner may seeinside the enamel and reconstruct the margin line. In addition, the useof additional wavelengths (e.g., green light) or even differentradiation modalities (e.g., ultrasound) imaging through the flesh may bepossible, allowing construction of margin lines and even teeth roots,and/or helping to distinguish structures such as dental caries from thedentin or other internal structures.

The resulting volumetric 3D model of the teeth may be used toreconstruct teeth base on the histological teeth. As described, thevolumetric model may be used to create dental prosthetics (implants,etc.) that have a more realistic appearance and/or a better fit.

Further, the methods and apparatuses described herein may permit a user(e.g., dentist, physician, dental technician, etc.) to follow the teethover time, including tracking dentin, caries, etc., and general dentalhealth by comparing models taken over time. For example, time-lapsevideos (images) may be constructed. FIG. 28A shows an example of avolumetric reconstruction taken at a first time, showing the dentin 3001(solid) and enamel 3003 (made slightly transparent). FIG. 28B showanother example of a volumetric model of teeth showing the dentin 3001and enamel 3003.

The volumetric model may include width information may provide estimatesof wear over time as well. For example, changes in the enamel width overtime and over different regions of the teeth may be easily tracked. Byknowing the enamel width we can estimate the tooth wear and provide asnap shot of the severity of wear.

Segmentation and Classification

Any appropriate method and/or apparatus (e.g., systems, devices,software, etc.) for generating images of internal structures from withina tooth (or other semi-transparent, strongly scattering object) may beused. For example, alternatively or additionally to the use ofscattering coefficients as discussed above, any of the apparatuses andmethods described herein may use the two-dimensional penetrative imagesalong with position and/or orientation information about the intraoralscanner relative to the object being imaged (e.g., the teeth) to segmentthe two-dimensional penetrative images and form a three-dimensionalmodel of the teeth including one or more internal structures within theobject. A penetrative image may refer to images taken with a near-IRand/or IR wavelength, revealing internal structures within the object(e.g., tooth). The position and/or orientation of the scanner may be aproxy for the position and/or orientation of the camera taking theimages which is on the scanner (e.g., on a handheld wand).

The apparatuses and methods described herein may construct athree-dimensional (3D) volumetric model of the teeth from segmentedtwo-dimensional (2D) images. These methods and apparatuses may alsosegment the 3D model of the teeth.

In general, the methods and apparatuses described herein allow for thedirect segmentation of the penetrative images. This may allow for theidentification of dentin within the teeth, including the location andmorphology of the dentin, as well as the identification and location ofcracks, lesions, and/or caries in the teeth, including in the dentin.The use of segmentation may allow for reconstruction of a volumetricmodel based on the penetrative images and the knowledge of the cameraposition corresponding to the penetrative images. A volumetric model ofteeth can be segmented and these segments (relating to differentinternal structures of the tooth) may be projected back to the imagesand/or combined with a surface model of the teeth (e.g., the outer toothsurface), allowing projections onto the surface images and bettersegmentation of the inner structures of teeth.

Thus, penetrative images taken through the teeth with a penetrativewavelength (e.g., near IR and/or IR), may include inner teeth structuresand/or 3D data. These images may be taken using any of the dentalscanners described herein, and the teeth volume may be segmented intodifferent regions according to opacity, color, and other properties ofthe images and 3D data. These regions can be for example: healthyenamel, dentin, lesion, dental filling(s), etc. The segmentation can bedone on 2D images or on volumetric models. The segmentation can be usedto classify the images and/or the 3D models according to the presence ofdifferent segments. A user may be able to detect by this segmentationmanually or automatically (or semi-automatically) to classify differentinternal structures, such as: dental caries, enamel erosion, and otherdental issues. Further, the images or models may be used to measureinternal regions of a tooth or multiple teeth segments for better dentaltreatments, including aligning teeth or other treatment planning. Forexample, a user may be able to locate dental lesion in an accuratefashion to plan accurate filling with minimal enamel extraction. Thus,the use of segmentation as described herein may permit the capture ofinner teeth structure without ionizing radiation, as is currently usedwith X-rays. Dental issues may be presented on 3D volumetric model.Further, as will be described in detail below, segmentation andclassification of internal structures may be automatized. Finally, exactmeasurements of internal structures may be taken for better treatmentplanning.

FIG. 17 illustrates an example of a data flow for scanning teeth with anintraoral scanner to build a 3D model including internal structures. InFIG. 17, the exemplary method shown may include three parts. First, theteeth may be scanned with an intraoral scanner 1701 (or any otherscanner) configured to provide penetrative scans into the teeth using anoptical (e.g., IR, near IR, etc.) wavelength or range of wavelengths.Any of these scanners may also concurrently scan to determine a surfacefeatures (e.g., via one or more non-penetrative wavelengths), color,etc., as described above. During scanning, a plurality of penetrativescans 1703, 1703′ may be taken, and the position of the sensor (e.g.,camera) 1705, 1705′ (e.g., x,y,z position and/or pitch, roll, yawangles) may be determined and/or recorded for each penetrative image. Insome variations, the surface of the teeth may also and concurrently beimaged, and a 3D surface model of the teeth 1707 determined, asdescribed above. In this example, the patient's teeth may be scanned,for example, with an intraoral 3D scanner 1702 that is capable ofimaging the inner teeth structure using, for example, near infra-redimaging. The location and orientation of the camera may be determined,in part, from the 3D scanning data and/or the 3D teeth surface model1707.

Thereafter, the penetrative images may be segmented 1711. In thisexample, segmentation may be done in one of two ways. On the inner teethstructure images, the images may be segmented using contour finding1713, 1713′. Machine learning methods may be applied to further automatethis process. Alternatively or additionally, near images (where theircamera position is close) may be used to decide on close features, andalso project features from the 3D model back to the images in order tolocate correctly segments like enamel. The method may also includeprojecting pixels from the inner teeth images back to the teeth andcalculating a density map of inner teeth reflection coefficient.Enclosing surfaces of different segments may be found or estimated byusing iso-surfaces or thresholds of the density map and/or by machinelearning methods. In addition, segmenting the images and projecting thesegments back to a model (such as the 3D surface model, e.g., projectingback to the world), may be used to find a segment by the intersection ofthe segment projections and the teeth surface.

The results may be displayed 1717, transmitted and/or stored. Forexample, the results may be displayed by the scanning system during theintraoral scanning procedure. The results may be shown by images withenclosing contours for different segments, a 3D density map, etc. In theexample shown in FIG. 17 a density map 1715, representing the dentinbeneath the enamel on the outer surface, is shown. This image may becolor coded to show different segments. In this example, internalsegments (structures) are shown within the 3D surface model (which isshown transparent); not all teeth have been scanned with penetrativeimages, thus, only some are shown. Alternative views, sections, slices,projections or the like may be provided. In FIG. 17, the example imageincludes artifacts that are present outside of the teeth 1716; these maybe removed or trimmed, based on the surface model 1718.

A segment may mark each pixel on the image. Internal structures, such asdentin, enamel, cracks, lesions, etc. may be automatically determined bysegmentation, and may be identified manually or automatically (e.g.,based on machine learning of the 3D structure, etc.). Segments may bedisplayed separately or together (e.g., in different colors, densities,etc.) with or without the surface model (e.g., the 3D surface model).

Thus, in FIG. 17, the patient is initially scanned with a 3D scannercapable of both surface scanning and penetrative scanning (e.g., near IRimaging), and the orientation and/or position of the camera is known(based on the position and/or orientation of the wand and/or the surfacescans). This position and orientation may be relative to the toothsurface. The method and apparatus may therefore have an estimate of thecamera position (where it is located, e.g., x,y,z position of thecamera, and its rotational position).

In general, penetrative images (e.g., near IR or IR images) may besegmented automatically. FIGS. 18A-18C illustrate a first example ofautomatic segmentation of a near-IR image. FIG. 18A, shows a firstautomatic segmentation of the outer surface of the teeth, determined by,e.g., edge detection. In FIG. 18A, the edges 1803 of the outer perimeterare shown. In this example, only a first level of edge detection wasperformed, looking for the outer perimeter. In FIGS. 18B and 18C, acontinuous edge region 1805 is shown, derived from the edge detection,and mapped onto the near-IR image (original image). FIGS. 19A-19C showthe identification and mapping of other edges from the same image. FIG.19A shows just the edges detected using a threshold setting value fromthe near-IR image (e.g., FIG. 19C). In FIG. 19B five (overlapping 1905)segments, 0-4, are traced from the detected edges by forming continuouslines. The different segments are shown color coded, and a color keyidentifying the segments is shown on the right. From the near-IR imagesthe apparatus can automatically segment the images. In FIGS. 18A-18C and19A-19C, the different segments are marked and may correspond todifferent regions (or different internal structures) on image. Whenmultiple images are analyzed, these putative segments may bere-projected back to a 3D model and/or shown in the images. FIGS.20A-20C and 21A-21C illustrate other examples of near-IR images from thesame patient shown in FIGS. 18A-19C, illustrating segmentation based onedge detection and identification of presumptive continuous line regionsfrom the detected edges. In FIG. 21A-21C, another region of the teethfrom the same patient are shown; eight segments (0-7) have beenidentified in this image, as shown in FIG. 21B. FIG. 21A shows the edgedetection of the original image, shown in FIG. 21C. FIGS. 22A-22Cillustrate segmentation of another region of the patient's teeth. FIG.22A shows the detected edges from the original near-IR image. FIGS. 22Band 22C show eight segments (0-7) identified on the near-IR image.Similarly, FIGS. 23A-23C illustrate segmentation of another region ofthe patient's teeth; FIG. 23A shows the detection of edges, FIG. 23Bshows segments identified from these edges, and FIG. 23C shows theoriginal near-IR image.

The segmented images, such as those shown in FIGS. 18A-23C may be usedto form a model of the internal structures of the scanned object (e.g.,teeth). The surface 3D model may also be used. For example, FIGS.24A-24B show a three-dimensional model of a region of the patient'steeth formed by segmented images, including those shown in FIGS.18A-23C. In FIG. 24A, the 3D reconstruction includes the outer surfaceof the teeth (shown as partially transparent), and different internalsegments may be shown in different colors and/or transparencies. Forexample, In FIG. 24A, the dentin (inner part of teeth) 2404 is shownwithin the teeth 2405 boundary. In FIG. 24A the segment showing thedentin is a surface (volume in FIG. 24B), but it may also be shown as adensity map, as will be illustrated in FIGS. 25A and 25B, below. Theresulting 3D volume including the segmented images may be iterativelyused to take images through the resulting volume, which may be‘projections’ that can be compared directly to the original near-IRimages, and this comparison may be used to modify the model. Thisprocess may be repeated (iterated) to refine the model, which mayprovide better segmentation of images.

As described above, segmentation may include edge detection. Anyappropriate edge detection method may be used, including machinelearning. Segmentation of the plurality of near-IR images may be used inconjunction with the positional information of the camera to reconstructthe volume. Since a plurality of different sections (different conics)are known, and segmented, the resulting segments inside of all of theprojections of the conics, from different positions are known andintersections of these segments may therefore be determined. Thisprocess may be made easier by using the outer surface boundary of theteeth, which may be provided by the surface imaging and/or the 3D model.As described above, this process may be iterative; the method may usethe 3D data to project simulated penetrative (e.g., near-IR) images thatmay be compared to the original to improve segmentation and derive asecond, evolved, model of the internal structures. Similarly, segmentsor segment regions outside of the teeth surface 2407 may be removed.

The model of the tooth, including internal structures, may be displayedin a variety of ways, as mentioned above. FIG. 24B shows a sectionthrough the teeth, showing the internal structures, including the dentin2404 and the enamel thickness between the outer surface 2405 and thedentin 2404.

FIGS. 25A and 25B show an reconstruction of the teeth including internalstructures (also shown in FIG. 17, above). In this example, the internalstructures are shown by a density mapping (e.g., segments). For example,the dentin 2505 is shown in more detail within a portion of the surfacemodel 2503 in FIG. 25B. The outer surface of the teeth may also beidentified as a segment (as shown in FIGS. 25A and 25B), and there isnear-perfect agreement between the segmented outer surface and the outersurface as determined from surface imaging in this example.

Sleeves for Intraoral Scanners Having Trans-Illumination

Any of the devices described herein may also include a sleeve or sleevesthat is configured to protect the intraoral scanner wand, but may alsobe configured to extend the functionality and/or adapt the scanner foruse with a penetrative wavelength, including trans-illumination. Thesleeve illustrated in FIGS. 29A-31B is an example of a sleeve that maybe used as a barrier (e.g., sanitary barrier) to prevent contaminationof the wand portion of the intraoral scanner, as the scanner may be usedwith different patients, and also as an adapter for providingtrans-illumination by IR/near-IR wavelength imaging. The sleeve in thesefigures is configured as a trans-illumination sleeve with electricalcouplings. For example, the sleeves described herein may include bothpenetrative wavelength illumination (e.g., near-IR and/or IR LEDs) andone or more sensors (e.g., CCDs) or may use the same cameras already onthe wand.

In FIG. 29A, the wand of an intra-oral scanner is shown with a sleeve3101 disposed around the end of the wand 3105; the sleeve is shown assemi-transparent, so that the internal structures (connectors) arevisible. FIG. 29B shows just the sleeve 3105 for the intraoral scanner(wand) shown as solid. In general, the sleeve 3105 slips over the end ofthe wand so that the light sources and cameras (sensors) already on thewand are able to visualize through the sleeve, and so that theelectrical contacts 3123, which may provide control, power and/or datatransmission to the LEDs and/or sensors 3125 integrated into or on thesleeve. The sleeve includes a pair of wing regions 3103 on oppositesides, facing each other and extending from the distal end of the wandwhen the sleeve is placed over the wand.

The sleeve 3101 may be held on the end of the wand by friction or by anattachment (not shown). Consequently, the sleeve may be readily removedfrom the wand and a new sleeve can be placed on the wand each time thescanner is used on a different patient. In this example, the sleeve maybe configured to transmit IR (e.g., near IR), and thus may include oneor more wings 3103 (e.g., for trans-illumination, etc.) as shown in FIG.29B. The electrical contacts and connector integrated into the sleevemay adapt the scanner for IR/near-IR trans-illumination.

Thus, the sleeve may include circuitry (e.g., flex circuitry) connectingto an LED illumination (IR/near-IR) source and/or one or more sensors,particularly for trans-illumination. FOR example, FIGS. 30A-30C. FIG.30A shows an example of the frame 3201 of the sleeve, which may be rigidor semi-rigid. The frame may support the flex circuitry 3203 (shown inFIG. 30B) and/or connectors 3205 and may also provide shielding (e.g.,blocking light). The frame and circuitry may be covered by a flexibleouter sleeve 3207 as shown in FIG. 30C.

The sleeve may be assembled by injection molding of the component parts,including the overall sleeve, windows for illumination and imagecapture, connectors for the circuitry and one or more LED holdingregions (e.g., injection of an IR and visible-light transparent materialforming windows through the sleeve, then injection of the rigid sleevematerial). The flex circuitry may then be positioned, and LEDencapsulation may be placed, using mold locators. The flexible outersleeve may then be injected.

FIGS. 31A-31C illustrate more detailed views of the flex circuitry 3301,connectors 3303 and LED holders/shields 3305. FIGS. 32A-32B illustrateexamples of the LED positioner and light blocker portion of the distalend of the sleeve. The example shown in FIG. 32A includes a supportframe or arm 3404 that extends down and includes a light shroud orblocker region 3406 encapsulating a portion of the LED. Exemplarydimensions are shown.

User Interface and Display of Volumetric Information

As mentioned above, the penetrative scans described herein may becollected from, for example, an intraoral scanner such as the oneillustrated in FIGS. 1A-1B for generating a three-dimensional (3D) modelof a subject's intraoral region (e.g., tooth or teeth, gums, jaw, etc.)which may include internal features of the teeth and may also include amodel of the surface, and methods of using such scanners. Although inmany instances surface scanning (including color scans) may be helpfuland useful, the penetrative (IR) scanning may, in some of the variationsdescribed herein, be sufficient.

A variety of penetrative scanning techniques (penetration imaging) maybe used or incorporated into the apparatuses described herein forperforming scans that to detect internal structures using a penetrativewavelength or a spectral range of penetrative wavelengths, including,but not limited, to trans-illumination and small-angle penetrationimaging, both of which detect the passage of penetrative wavelengths oflight from or through the tissue (e.g., from or through a tooth orteeth). Thus, these apparatuses and techniques may be used to scanintraoral components such as a tooth or one or more teeth, gingiva,palate, etc. and used to generate a model of the scanned area. Thesemodels may be generated in real time or after scanning. These models maybe referred to as 3D volumetric models of the teeth, but may includeother regions of the jaw, including the palate, gingiva and teeth.Although the methods and apparatuses described herein typically relateto 3D volumetric models, the techniques and methods described herein mayalso be used in some instance with 3D surface models. The surface modelinformation is typically part of the 3D volumetric model.

The collection and analysis of volumetric data from the intraoral cavitymay identify features and information from teeth that were previouslydifficult or impossible to identify from non-volumetric scanning.However, it may be difficult or non-intuitive for a dental practitioner(and/or patient) to analyze three-dimensional volumetric information.Described herein are methods and apparatuses for viewing andinterpreting 3D, volumetric data of a patient's oral cavity.

For example, FIGS. 38A-38G illustrate one example of a method fordisplaying 3D volumetric data. FIG. 38A shows a surface model (which maybe a surface model portion of a volumetric model) from a top view of anupper arch, in which external features are visible (e.g., surfacefeatures). This view is similar to the surface scan view which may be incolor (e.g., taken by visible light). Internal structures, which arepresent within the model beneath the external surface of the scan, arenot readily visible in FIG. 38A. FIG. 38B, the internal structures areshown based on their relative transparency to near-IR light. In FIG.38B, the enamel is more transparent (and is shown as more transparent)than the dentin, which is shown as less transparent. FIGS. 38B-38Fillustrate a transition between the surface view of FIG. 38A and thepenetrative, internal 3D view of FIG. 38B for a sub region (circledregion “C”) shown. For example, a user display may be provided in whichthe relative surface vs. internal views may be altered to provide asense of internal structures within the dental arch relative to surfacestructures. For example, an animated view cycling through image such asFIGS. 38C-38G may be provided. Alternatively, the user may slide aslider 3803 toggling between the surface and internal views. Thetransition between these two views (which may be made from any angle,may help the user and/or patient to see beneath the surface of theteeth, to visually assess the rich internal data. The 3D volumetricmodel may be manipulated to show any view, including cross-sectionalviews, showing internal structures and/or surface features. In FIG.38A-38G the top view is shown. FIGS. 38C-38G illustrate progressivelymore transparent views or a region (“C”) of the 3D volumetric model ofFIG. 38A in which progressively large percentages (from 0% to 100%) ofthe internal view of FIG. 38B is shown for region C, while progressivelyless of the surface view (from 100% to 0%) is shown.

FIGS. 40A-40C illustrate another example, showing a hybrid image that(like FIG. 38E combines and mixes both surface image scanning (e.g., avisible light scan, as shown in FIG. 40A) with a volumetric model takenusing a penetrative (e.g., near-IR) wavelength, as shown in FIG. 40B.Features that are present in the tooth enamel and dentin are visible inthe volumetric reconstruction (image shown in FIG. 40B) that are notapparent in the image (which may also be a reconstruction) of just thesurface shown in FIG. 40A. For example, in FIG. 40B, a carries region1103 is apparent, which is not visible in FIG. 40A. Similarly a bubbledregion of the enamel 1105 is visible in FIG. 40B but is not visible inFIG. 40A. FIG. 40C shows a hybrid image of the 3D volumetric model andsurface model (surface image), in which both of these structures, thecarries and the bubbled region, are visible.

In general, described herein are methods and apparatuses for simplifyingand displaying volumetric data from a patient's oral cavity (e.g.,teeth, gingiva, palate, etc.) in a manner that may be easily understoodby a user (e.g., a dental practitioner) and/or a patient. Also describedherein are methods of displaying volumetric data taken from a patient'soral cavity in a manner that may be familiar for a user and/or patientto understand. In a first example, the data may be presented as one or aseries of x-ray type images, similar from what would be produced bydental x-rays. FIG. 33 illustrates one method of generating x-ray (orpseudo x-ray) images from a volumetric data set taken as describedabove, e.g., using penetrative light (e.g., near-IR) wavelength(s).

As shown in FIG. 33, a method of displaying 3D volumetric images of apatient's oral cavity may include receiving the 3D volumetric data 3301,e.g., from a scan as described above, either directly or from a storeddigital scan, etc. In some variations, individual teeth or groups ofteeth may be identified from the volumetric data 3303. The teeth may beidentified automatically (e.g., by a segmenting the volume, by machinelearning, etc.) or manually. Alternatively, the entire volume may beused. Pseudo-x-ray images may then be generated from the volume orsub-sets of the volume corresponding to individual teeth 3305. Forexample, an image of the volume may be taken from the ‘front’ of thetooth or teeth, in which the transparency of the enamel (and/orenamel-like restorations), dentin and other features are kept from thevolumetric data. This volumetric data may be based on the absorptioncoefficients of the material within the oral cavity for the penetratingwavelength of light used. Thus, a projection through the volumetric datamay be generated for a fixed direction from the volumetric data to getan image similar to an X-ray, but, in some variations, inverted andshowing the density of the dentin (highly absorbing) as “darker” thanthe density of the enamel (less absorbing and therefore moretransparent); caries may also show up as more absorbing (darker)regions. The image may therefore be inverted to resemble an x-ray imagein which more dense regions are lighter (e.g., brighter). These pseudox-ray images may be generated from the same positions as standard dentalx-rays and presented to the user. For example, a panel of pseudo x-rayimages may be generated from the volumetric model for each of thepatient's teeth. Although the penetration of the wavelength of the light(e.g., near IR light) may not be as deep as with traditional x-rays,images generated in this manner may provide a comparable proxy for anx-ray, particularly in the crown and mid-tooth regions above thegingiva.

Other simplified or modified displays may be provided to the user, orcustomized for display by the user to the patient. For example, in somevariations images of the teeth may be generated from the volumetric datain which the image is simplified by pseudo coloring the volumetric datato highlight certain regions. For example, regions that have beenpreviously marked or flagged (as will be described in greater detail,below) may be colored in red, while the enamel may be shown as a morenatural white or slightly off-white color. In some variations,enamel-like materials (e.g., from fillings, etc.) may be representedseparately and/or marked by a color, pattern, etc.

In some variations, the methods and/or apparatuses may display the teethin sections through the dental arch. Similarly the individual teeth orgroups of teeth may be shown separately and/or labeled (e.g., bystandard naming/numbering convention). This may be shown in addition orinstead of other displays. In some variations, the teeth and/or internalstructures may be pseudo-colored or projecting on to a color image maybe used.

For example, FIGS. 34A and 34B illustrate virtual sections taken througha volumetric model of a patient's teeth generated from an intraoral scanthat included near-IR information, and described above. In FIG. 34A, thecross-sectional view may be generated automatically or manually, e.g.,by the user, to display regions of interest within teeth, includingenamel. The cross-section may show both density sectioning and/orsurface sectioning. These images may be pseudo-colored to show differentregions, including outer surfaces, enamel, dentin, etc. Internalstructures, e.g., within the enamel and/or dentin, may reflect theeffect of the near-IR light within the teeth, such as the absorptionand/or reflection of light at one or more near-IR/visible wavelengthswithin the teeth. In FIG. 34B, the section is pseudo-colored with a heatmap to show internal features, and a key may be provided, as shown. Inany of these variations, 2D projections of the teeth may be generatedfrom the volumetric information, showing one or more features on thetooth and/or teeth. As will be described in greater detail below,additional features, including lesions (e.g., caries/cavities, cracks,wearing, plaque buildup, etc.) may be displayed as well, and may beindicated by color, texture, etc. While illustrated as sections of the3D volumetric model, other embodiments may display the 2D section byitself to provide a cross-sectional view of the tooth/teeth similar to aview provided by 2D x-ray images.

Any of the methods and apparatuses for performing them described hereinmay include displaying one or more (or a continuous) sections through a3D model of the patient's dental arch, and preferably a 3D volumetricmodel. For example, a method of displaying images from athree-dimensional (3D) volumetric model of a patient's dental arch mayinclude: collecting the 3D volumetric model of the patient's dentalarch, wherein the 3D volumetric model includes near-infrared (near-IR)transparency values for internal structures within the dental arch;generating a two-dimensional (2D) view into the 3D volumetric modelincluding the patient's dental arch including the near-IR transparencyof the internal structures; and displaying the 2D view. In any of thesemethods, the method may optionally (but not necessarily) includescanning the patient's dental arch with an intraoral scanner.

Generating the 2D view may comprises sectioning the 3D volumetric modelin a plane through the 3D volumetric model. The user may select theplane's location and/or orientation, and my do this in a continuousmanner. For example, any of these methods may include selecting, by auser, a section though the 3D volumetric model to display, whereinselecting comprises continuously selecting sections through the 3Dvolumetric model as the user scans through the 3D model and continuouslydisplaying the 2D views corresponding to each section. Generating the 2Dview may comprises selecting, by a user, an orientation of the 2D view.

In any of these methods, the surface may be included. For example, asdescribed and illustrated above, a method of displaying images from athree-dimensional (3D) volumetric model of a patient's dental arch mayinclude: collecting the 3D volumetric model of the patient's dentalarch, wherein the 3D volumetric model includes surface values andnear-infrared (near-IR) transparency values for internal structureswithin the dental arch; generating a two-dimensional (2D) view into the3D volumetric model including the patient's dental arch including bothsurface values and the near-IR transparency of the internal structures;and displaying the 2D view. The surface values may comprise surfacecolor values. The surface relative to the internal (volumetric)structures may be weighted. For example, generating the two-dimensional(2D) view through the 3D volumetric may also include including in the 2Dview a weighted portion of the surface values and a weighted portion ofthe near-IR transparency of the internal structures. The weightedportion of the surface values may include a percentage of the full valueof the surface values, and the weighted portion of the near-IRtransparency of the internal structures comprises a percentage of thefull value of the near-IR transparency of the internal structures,wherein the percentage of the full value of the surface values and thepercentage of the full value of the near-IR transparency of the internalstructures adds up to 100%. For example, the user may adjust theweighted portion of one or more of the surface values and the near-IRtransparency of the internal structures.

For example, a method of displaying images from a three-dimensional (3D)volumetric model of a patient's dental arch may include: collecting the3D volumetric model of the patient's dental arch, wherein the 3Dvolumetric model includes surface color values and near-infrared(near-IR) transparency values for internal structures within the dentalarch; selecting, by a user, an orientation of a view of the 3Dvolumetric model to display; generating a two-dimensional (2D) view intothe 3D volumetric using the selected orientation, including thepatient's dental arch including a weighted portion of the surface colorvalues and a weighted portion of the near-IR transparency of theinternal structures; and displaying the 2D view.

In addition to displaying qualitative images of the teeth, the methodsand apparatuses described herein may quantify, and may providequantitative information about internal and/or external features. Forexample, volumetric measurements of one or more lesions may be provided(selectably or automatically) including dimensions (peak or mean length,depth, width, etc.), volume, etc. This may be performed by manually orautomatically segmenting the volumetric model to define the regions ofinterest, including either or both tooth features (enamel, dentin, etc.)and/or irregularities (e.g., caries, cracks, etc.). Any appropriatesegmentation technique may be used, such as but not limited to: meshsegmentation (mesh decomposition), polyhedral segmentation,skeletonization, etc. Once the volume has been segmented, these regionsmay be separately or collectively displayed and/or measured. As will bedescribed below, they may also be marked/flagged and used for furtheranalysis, display and modification of the scanning methods and systems.

In some variations of the user interfaces described herein, a summaryreport may be generated or created and displayed for the user and/orpatient from the volumetric data. For example, summary data may beprojected onto a model of the patient's teeth. The model may also besimplified, so that the enamel is opaque, but marked or selectedinternal features (including automatically selected internal features)are shown in red or some other contrasting color (and/or flashing,blinking, etc.) within the tooth. For example, caries may be shown inthis manner. The summary report may be automatically entered into apatient chart.

Any of the images, including the volumetric images, may be animated. Forexample, virtual sections through the patient's teeth, showing ascanning or traveling cross-section through the patient's dentition maybe shown, in some cases with a 3D model showing one or more cutting axesthrough the volume. The user interface may allow the user to section inone or more planes, showing both external and internal features based onthe volumetric scan.

In general, the apparatuses described herein may generate separate viewfor the user (e.g., physician, dentist, orthodontist, etc.) than thepatient. The user may be provided with a ‘clinical view’ that mayinclude information not present on a separate ‘patient view.’ Theclinical view may be more technical, and may in some cases be closer tothe raw images from the volumetric data. The user may select whichlayers of information to include in the patient view, which may then bepresented to the user during or after the scanning or review of thedental scanning. Patient educational materials may be appended to thepatient view.

For example, in some variations, the user display of volumetric data mayinclude an overlay of the volumetric data in which pseudo coloring ofthe 3D components within the volumetric data is shown. As will bediscussed in more detail below, in any of these displays/images markedor highlight regions may be shown to call attention to potential problemregions (e.g., caries, thin enamel, cracks, etc.). Two-dimensional (2D)color data and 3D near-IR data (e.g., surface and volumetric regions)may be shown, including transitions between the two.

In general, the volumetric information may be annotated (e.g., marked,labeled, etc.) either automatically, manually, or semi-automatically,and this annotation may be displayed. Furthermore, annotations may beused both to annotate future additional scans, and to modify how futurescans of the same patient are taken and displayed. An annotation may be,for example, a marker or flag on a region of interest. Regions ofinterest may correspond to specific regions in which one or morefeatures (cracks, caries, thinning of enamel, buildup of plaque orcalculus, etc.) have been observed. Alternatively or additionally,regions of interest may be regions in which there is a change over time,e.g., from one scan to another scan.

As mentioned above any of these methods may include placing one or moremarkers on the volumetric model of the patient's teeth. Markers (e.g.,flags, pins, etc.) may be manually placed by the user, or may beautomatically placed by the apparatus, or may be semi-automaticallyplaced (e.g., suggested by the system, configured by the user, etc.).This is described in greater detail below.

Markers may be used to focus attention and/or processing by the systemon one or more specific regions of the volumetric model for display,and/or for later follow-up (e.g., in future scans). Markers may modifythe manner in which the later scans are taken, e.g., taking future scansof marked regions with greater detail (e.g., higher resolution,different wavelengths, greater scanning frequency or reputations, etc.).Marked regions may be displayed over time to show changes in the markedregions.

For example, a user can mark a digital representation of the patient'steeth (or the patient's actual teeth) with a marker (e.g., a pin, flag,etc.) which can be annotated (e.g., can have notes associated with it).This marker may then be used to track over time between different scans.Later scans can be marked in the corresponding location, the later scancan be modified based on the marked regions. These marked regions may bescanned in greater detail, and analytics may be automatically performedand/or displayed, measuring and/or indicating a change compared to oneor more earlier scans. Thus any of the systems described herein maytrack one or more marked regions from previous scans and give feedbackduring and/or after a new scan, providing additional detail. This can bedone for both surface and/or volumetric information, particularly on theproperties of the enamel, and/or by comparison to the enamel, the outersurface of the teeth/tooth, and/or the dentin.

For example, one or more annotation markers from an earlier scan maymodify a subsequent scanning of the same patient. Before scanning, theuser may enter an identifier of the patient being scanned (alternativelythe system may automatically identify the patient based on a database ofearlier scans). The system may automatically annotate the new scan basedon the prior scan annotations.

In some variations the later scans may be automatically annotated by thesystem by identifying differences between the prior scan(s) and thecurrent scan. For example, regions showing a change above a thresholdcompared to the earlier scan may be flagged and presented to the user.The annotation may be done without user oversight (fully automatic) ormay be done with some user oversight, for example, by flagging it andindicating to the user why it was flagged, then allowing the user tokeep, modify or reject the marking. Reasons for automatically markingthe teeth may include a change in the enamel thickness, a change in thesurface smoothness, a change in the relative ratio of enamel vs. dentinin a tooth, a change in the position of the tooth (e.g., occlusion),etc. It may also include a change in structures external to the naturaltooth, such as increase or decrease in plaque or calculus buildup, orchanges to the gingival structures surrounding the tooth, Thus, if thesystem detects one or more of these conditions, it may automaticallyflag the relevant region in the volumetric model.

Later scans may be dynamically modified by the flags from earlier scansor by a detection of a change in a region (even unmarked regions)compared to earlier scans. For example, the scanning parameters may bemodified to scan at higher resolution (e.g., changing the scan dwelltime, requiring the user to scan this region multiple times, etc.),changing the wavelength used for the scanning, etc.

For example, FIG. 35 illustrates a method of automatically selecting aregion for marking and/or using the selected regions. A first volumetricmodel of the patient's teeth is generated from a scan of the patient'steeth 3501. The volumetric model may be generated using any appropriatemethod, including those described above and discussed in U.S. patentapplication Ser. No. 15/662,234, filed Jul. 27, 2017, titled “INTRAORALSCANNER WITH DENTAL DIAGNOSTICS CAPABILITIES”, incorporated by referencein its entirety. The first volumetric model may be stored (digitallystored) as part of the patient's dental record. The first volumetricmodel may concurrently or subsequently (immediately or some timethereafter) be analyzed (e.g., by an apparatus, including an apparatushaving a processor that is configured to operate as described herein) toidentify any regions that should be flagged 3503. This analysis maytherefore be performed automatically, and may examine one or moreproperties of the patient's teeth from the scan. Automatic (orsemi-automatic, etc., automatic but with manual assistance toverify/confirm) may be performed by a microprocessor, including systemsthat have been trained (e.g., by machine learning) to identify regionsof irregularities on the outside and/or internal volume of the teeth.For example, the apparatus may examine the digital model to identifypossible defects in the patient's teeth such as (but not limited to):cracks, caries, voids, changes in bite relationship, malocclusions, etc.This may include identifying regions in which there is an optical (e.g.,near-IR) contrast near the surface of the tooth indicating a possiblecrack, caries, occlusion, etc. 3505. Regions that are less transparent(e.g., more absorptive) in the near-IR wavelengths than the rest of theenamel, that are closer to the surface (generally or within specificnear-IR wavelengths), may correspond to defects. Alternatively oradditionally, surface properties of the teeth may be examined andflagged if they are outside of a threshold 3507. For example, regions ofthe surface of the teeth in which the tooth surface is rough (e.g., hasa smoothness that is below a set threshold, where smoothness may bedetermined from the outer surface of the enamel) may be flagged. Othersurface properties may also be analyzed and used to determine if theregion should be marked or presented to the user to confirm marking,including discoloration (based on a color or white-light/surface scan),gingival position (relative to the outer surface of the tooth), etc. Thedistribution and size of the patient's enamel may also be examined 3509.The enamel thickness may be determined from the optical properties(e.g., comparing absorption/reflectance properties). Regions of putativeenamel that are below a threshold thickness, or having a ratio ofthickness to tooth dimension (e.g., diameter, width, etc.) below athreshold may be marked or presented to the user to confirm marking.

In some variations, during and/or after automatically analyzing thevolumetric model, the use may also manually flag one or more regions ofthe volumetric model of the patient's teeth 3515. If the automaticanalysis of the volumetric model automatically flags the identifiedvolumetric model the user's manually added regions may be added. In somevariations, the user may be prompted to flag the regions identified andsuggested by the automatic analysis. These regions may be marked and anindication of the reason(s) for their being identified may be provided(e.g., irregular enamel, potential crack, potential caries, potentialthinning of the enamel, etc.). In general, the inner boundary (theboundary within the volume of the tooth, for example) may be defined inany of the methods and apparatuses described herein. For example, invariations in which a region of the enamel is thinning, the methods andapparatuses described herein may be used to the entire layer (e.g.,layer of enamel, region of the inner structure of the tooth) may beidentified and used for qualitative and/or quantitative information.

In some variations the method flagged regions may then be displayed on adigital model of the patient's teeth 3517. The display may emphasize theflagged regions, e.g., by providing a color, animation (e.g., flashing),icon (e.g., arrow, flag, etc.), or the like, including combinations ofthese. The display may also show enlarged views of any of these. Theuser may modify the display, e.g., rotating, sectioning, enlarging,etc., the flagged region. Alternatively or additionally, the flaggedregions may be enlarged on the display by default. An index or key ofthe flagged regions may be provided, and may be displayed and/or storedwith the digital volumetric model of the patient's teeth.

In some variations, as shown to the right of FIG. 35, the method mayinclude using the flagged regions to modify the future scans, asmentioned above. For example, the method may include scanning(“rescanning”) using the flagged regions, after some interim time periodof between a first time (e.g., about one day, one week, one month, 2months, 3 months, 4 months, 5 months, 6 months, 7 months, 8 months, 9months, 10 months, 11 months, 1 year, 1.2 years, 1.5 years, 2 years,etc.) and a second time (e.g., about one month, 2 months, 3 months, 4months, 5 months, 6 months, 7 months, 8 months, 9 months, 10 months, 11months, 1 year, 1.5 years, 2 years, 3 years, 4 years, 5 years, 6 years,7 years, 8 years, 9 years, 10 years, 11 years, 12 years, 13 years, 14years, 15 years or more, etc.), or longer than the second time period.The flagged regions may be used to modify the scan by increasing theresolution of the scanned regions during the scan, e.g., by increasingthe scan rate, increasing the dwell time in this region, scanning theregion in additional wavelengths, scanning this region multiple times,etc. The scanning apparatus may inform the user to adjust the scanning(e.g., moving the wand of an intraoral scanner more slowly in theseregions, moving the scanner back over these regions multiple times,etc.) and/or it may automatically adjust the scanning parameters duringoperation. The scanning apparatus may therefore receive the key or indexof flagged regions and/or a marked (flagged) version of the patient'searlier intraoral scan(s). Prior to scanning the scanning apparatus, theuser may indicate the identity of the patient being scanned, and thismay be used to look up the earlier scan(s). Alternatively oradditionally, the apparatus may identify the patient based on thecurrent scan to identify (or confirm the identity) of the patient, toverify or recall the earlier annotated (flagged) scan. Alternatively,the second or subsequent scans may be taken without using the earlierflagged regions.

Following the subsequent (e.g., second, later or follow-up scan orscans), the method or apparatus configured to perform the method maythen compare the flagged regions from the subsequent scan with thecorresponding regions from the earlier scan(s) 3521. In addition, thevolumetric model from the subsequent scan may automatically analyzed toidentify any regions of the new (subsequent) scan that can/should bemarked/flagged (e.g., repeating the earlier automatic or semi-automaticanalysis steps 3503-3515) 3521. Newly identified regions from thesubsequent scan may be compared to the previously un-flaggedcorresponding regions in the earlier volumetric model(s).

The flagged regions may be analyzed over time 3523. Specific sub-regionsfrom the volumetric model including the flagged regions may be generatedfor display and analysis. The results may be output 3525. For example,these regions may be displayed to the user along with descriptive,analytic information about the scanned region. These regions may also bemarked to shows changes over time. The data may be displayed in ananimation, for example, showing changes over time. In some variations,the images may be displayed as a time-lapse image (video, loop, etc.),showing changes. Time-lapse images may show the change in the internaland/or external structure over time. Sections (pseudo-sections generatedfrom the volumetric model(s)) may be used to show changes. Color,texture, patterns, and any other highlighting visualization techniquemay be used. Alternatively or additionally to the display of the flaggedregions, these regions (and any accompanying analysis) may be output inother appropriate ways, including digitally outputting (e.g., thepatient's dental record), printing a description of the flaggedregion(s), etc.

Any of the methods of tracking a region of a patient's dental archdescribed herein may include tracking over time and/or across differentimaging modalities (e.g., records) as described in greater detail below.Further, any of these methods may be automated and/or may includeautomated agents, for example, for identifying one or more regions ofinterest (e.g., features, defects, including actionable dentalfeatures), including for scoring them and/or automating identification,scoring and/or display of such regions. Any of these methods may alsoinclude any of the display methods or agents (e.g., for displayingsections, displaying internal structures, for displaying virtual x-rays,for displaying across imaging modalities, etc.).

For example, a method of tracking a region of a patient's dental archover time may include: collecting a first three-dimensional (3D)volumetric model of the patient's dental arch, wherein the 3D volumetricmodel includes surface values and near-infrared (near-IR) transparencyvalues for internal structures within the dental arch; identifying aregion of the 3D volumetric model; flagging the identified region;collecting a second 3D volumetric model of the patient's dental arch;and displaying one or more images marking, on the one or more images, adifference between the first 3D volumetric model and the second 3Dvolumetric model at the flagged region.

Any of these methods may also include tracking and/or comparing acrossdifferent records (e.g., different imaging modalities), so thatidentifying comprises identifying a region of the patient's dental archfrom a first record of a plurality of records, wherein each recordcomprises a plurality of images of a patient's dental arch each takenusing an imaging modality, further wherein each record of the pluralityof records is taken at a different imaging modality, further whereinflagging comprises flagging the identified region in a correspondingregion of the 3D volumetric model of the patient's dental arch. Themethod and apparatuses for performing them may also include correlatingthe flagged region with each of records of the plurality of records bycorrelating the 3D volumetric model of the patient's dental arch witheach of the records of the plurality of records. In some variations thecorrelation may be used to weight or grade the identified region todetermine if it corresponds to a region of interest (e.g., a feature, adefect, including actionable dental features, etc.). For example, theregion of the patient's dental arch may comprises a dental featurecomprises one or more of: cracks, gum recess, tartar, enamel thickness,pits, caries, pits, fissures, evidence of grinding, and interproximalvoids. Identifying the region may comprise comparing a near-IRtransparency value of a region within the 3D model to a threshold value.

Where surface values are used, the surface values may comprise surfacecolor values. These methods may be used with stored data and/or withdata collected in real time (e.g., thus the method may optionally butnot necessarily collect a three-dimensional (3D) volumetric model byscanning the patient's dental arch to generate the 3D volumetric model.

Identifying the region may comprise comprises automatically identifyingusing a processor. For example, automatically identifying may compriseidentifying a surface color value outside of a threshold range.Automatically identifying may comprise segmenting the 3D volumetricmodel to identify enamel regions and identifying regions having enamelthicknesses below a threshold value.

Flagging the identified region may comprise automatically flagging theidentified regions or manually confirming the identified region forflagging.

In any of these method in which regions are flagged, the method mayinclude re-scanning the patient's dental arch wherein the flagged regionis scanned at a higher resolution than un-flagged regions.

A method of tracking a region of a patient's dental arch over time mayinclude: collecting a first three-dimensional (3D) volumetric model ofthe patient's dental arch taken at a first time, wherein the 3Dvolumetric model includes surface color values and near-infrared(near-IR) transparency values for internal structures within the dentalarch; identifying, using an automatic process, a region within the 3Dvolumetric model to be flagged from a first record of a plurality ofrecords, wherein each record comprises a plurality of images of apatient's dental arch each taken using an imaging modality, furtherwherein each record of the plurality of records is taken at a differentimaging modality; flagging the identified regions; correlating theflagged region with each of the records of the plurality of records bycorrelating the 3D volumetric model of the patient's dental arch witheach of the records of the plurality of records; collecting a second 3Dvolumetric model of the patient's dental arch taken at a separate time;and displaying a difference between the first 3D volumetric model andthe second 3D volumetric model at the flagged region.

Similarly, as summarized and described above, a method of tracking adental feature across different imaging modalities, the methodcomprising: collecting a first three-dimensional (3D) volumetric modelof the patient's dental arch, wherein the 3D volumetric model of thepatient's dental arch includes surface values and internal structureswithin the dental arch; identifying a region of the patient's dentalarch from a first record of a plurality of records, wherein each recordcomprises a plurality of images of a patient's dental arch each takenusing an imaging modality, further wherein each record of the pluralityof records is taken at a different imaging modality; flagging theidentified region in a corresponding region of the 3D volumetric modelof the patient's dental arch; correlating the flagged region with eachof the records of the plurality of records by correlating the 3Dvolumetric model of the patient's dental arch with each of the recordsof the plurality of records; and saving, displaying and/or transmittingimages including the region of the patient's dental arch. Any of thesemethods may include tracking over time as well, e.g., by comparing thesame region(s) to 3D volumetric models at different times.

FIGS. 39A and 39B illustrate a user interface showing marking of aregion of interest from a 3D volumetric scan of a patient's oral cavity.In FIG. 39A, the user interface includes an image of the internalfeatures 3901 (e.g., based on the near-IR absorption of the teeth),similar to FIG. 38B, discussed above. This view may be manipulated byuser controls 3915, including sectioning tools, rotation, moving tools,etc. In FIG. 39A, two upper windows show a surface view 3903 and avolumetric (internal) view 3905 corresponding to the same region. Thisregion may be selected. FIG. 39B shows the same features of FIG. 39A,but with a region marked or flagged 3911, 3911′. As discussed above, theidentification of the region to be flagged may be automatic or manual,or semi-automatic (e.g., confirmed by the user), and may be chosen toselect a region for later monitoring. In FIG. 39B, the region maycorrespond, for example, to possible caries.

Monitoring of one or more internal regions of the teeth over time usingthe volumetric models of the patient's teeth taken with the devicesdescribed herein may be particularly helpful for predicting dentalproblems, including caries, cracks, tooth loss, gum recession, and thelike. In particular, these methods and apparatuses may help the user(e.g., dentist, dental technician, orthodontist, etc.) inform andeducate a patient so that the patient may take recommended treatmentsprior to developing more serious problems. There is a need for effectiveways to show changes in teeth over time and to provide patients withinformation necessary to act early to prevent more complicated andpotentially painful problems from developing. Many patients areotherwise reticent to undergo preventive procedures, particularly whenthere is not currently any associated pain or discomfort. For example,pre-cavitated caries are difficult to identify with current imagingtechniques, and it may be particularly difficult to convince a patientto treat even when identified, since they typically present withoutpain. However, early stage treatment may be critical to avoiding morecomplicated and dangerous procedures later.

Dental caries are one type of problem that may be identified with themethods and apparatuses described herein. As shown and discussed above,caries may be identified from 3D volumetric models (such as thosedescribed herein) that penetrate into teeth using light (e.g., near-IR),one type of non-ionizing radiation. In the 3D volumetric modelsgenerated as described herein, e.g., using a near-IR light, typically incombination with a surface scanning (e.g., white light), the absorptioncoefficients of the internal regions of the teeth may indicatedistinctions between dentin and enamel, and may reveal internalstructures and flaws, including cracks, caries and the like. Forexample, regions of enamel that are less transparent than expected inthe near-IR wavelengths may (e.g., having different IR opticalproperties), and particularly those that appear to extend to the surfaceof the tooth in the volumetric model may be identified manually orautomatically as cavities or caries. Other irregularities in the enameland/or dentin (e.g., based on the internal features of the teeth fromthe volumetric model) may be identified and may be characteristic of aproblem in the teeth. Thus, the techniques described herein may be usedfor prognosis of dental issues such as caries.

As mentioned, any of the apparatuses and methods described herein mayinclude improved methods for displaying of internal tooth features usingthe one or more volumetric models of a patient's teeth. For example, themethods and apparatuses described herein may be used to generate anestimate of enamel thickness for one or more of the patient's teeth.These estimates may be visually displayed, showing the outer surface ofthe teeth or a particular tooth, and may also show internal structures,including in sectional views or 3D internal views showing, e.g., theenamel, including the thickness of the enamel. This information may beused clinically to determine the need for, to help design and to helpapply dental prosthetics, including veneers, crowns, and the like. Anyof the methods and apparatuses described herein may be used, forexample, to help prepare design a dental implant for a particular toothor teeth.

Plaque and Calculus Detection and Visualization

The method and apparatuses described herein may also be used for thedetection and visualization (including quantification) of plaque andcalculus on the patient's teeth. Any of the intraoral scanners describedherein may be used to detect plaque or calculus on the patient's teethby using florescence imaging in addition to other imaging/scanningmodalities including penetrative (e.g., near-IR) imaging. For example,and intraoral scanner may be cycled between different imaging modalitiessuch as between white light and near-IR, including additional modalitiessuch as florescence (e.g., laser florescence, etc.).

The use of fluorescence capabilities (and/or using current ones) by theintraoral scanner may allow detection of plaque and calculus on thesurface of the teeth. In combination with 3D modeling using the datafrom the intraoral scanner, the plaque/calculus conditions can bemodeled and visualized on the 3D model of the teeth, including the 3Dvolumetric modeling of the teeth. Plaque and/or calculus may be detectedand may be displayed and highlighted as described above, and may be usedbefore, during or after treatment. For example, a dental technician(e.g., dental hygienist) may use an intraoral scanner to detect andmonitor the condition of the patient and the cleaning treatment. Data onplaque and calculus may also be used by any of the apparatuses describedherein to determine and provide predictive models that may indicateplaque and calculus (e.g., tartar) generation rate and/or locations.

In some variations, plaque and calculus may be identified at least inpart using florescence information. It has been observed that plaque mayfluoresce under blue light (e.g., around about 405 nm). Any of theintraoral scanners described herein may include fluorescence informationfrom which information about plaque and calculus may be used, andincorporated into a 3D model of the patient's teeth. For example, plaqueand/or calculus may be visually displayed as a color and/or texture onthe 3D model of the patient's teeth.

For example, a fluorescence signal can be obtained from an intraoralscanner using a dichroic filter having a large aperture amplification offluorescence signal. This amplification may emphasize the fluorescence,thus enabling the detection, visualization and segmentation of plaqueand calculus regions using RGB illumination, sensor and image.Alternatively or additionally, the apparatus may include a florescencesource (e.g., an LED emitting at 405 nm) and corresponding filter(s) fordetection of plaque and/or calculus. This may be integrated in theintraoral scanner, or added (e.g., as a sleeve, accessory, etc.) to beused with the scanner.

Alternatively or additionally, in some variations, depending on thewavelength of near-IR light used, the plaque and calculus may have adifferent absorption/reflection than enamel. This may allow the calculusand/or plaque to be differentiated from the enamel in the volumetricmodel. Further, the volumetric model may be used to detect material onthe teeth, including calculus and plaque based on the surface smoothnessand geometry. In variations in which calculus and/or plaque are nottransparent to the near-IR frequencies used, the apparatus maydifferentiate calculus and/or plaque from the enamel using thevolumetric model. Thus, the calculus and/or plaque may be segmented anddifferentiated from enamel.

The use of an intra-oral scanner to detect plaque and/or calculus mayprovide quantitative information and digital modeling. This may allowmonitoring and comparison of plaque/calculus over time based onregistration to 3D model, including real-time registration and/ordisplay.

The acquisition of both fluorescence image and 3D scan on the same timeand same position of the intraoral scanner (e.g., the scanning wand)allows for very accurate registration of the plaque/calculus regions andthe 3D model. The concurrent scanning is described in greater detail,for example, in U.S. patent application Ser. No. 15/662,234, filed Jul.27, 2017, and titled “INTRAORAL SCANNER WITH DENTAL DIAGNOSTICSCAPABILITIES”. The accurate registration between different scanningmodalities, such a white/visible light, penetrative (near-IR) lightand/or florescence, may enable the apparatuses to define the borders ofthe calculus and/or plaque and may permit the apparatus to determine thevolume/thickness in high resolution, allowing for both measuring theprecise current situation and comparison/tracking relative to previousscans.

The methods and apparatuses described herein may take RGB images of theteeth at the same/similar time with taking 3D scans of the teeth. Thesescans may then be used to build the 3D model of the teeth/jaw, which mayinclude the volumetric information (3D volumetric model). For example,RGB images may show emphasized signal of fluorescent surfaces,specifically plaque and calculus regions, due to specific characteristicof color and brightness of such surfaces, as mentioned. For example, theimage of the outer surface (and in some cases the volumetric model) ofthe teeth may show regions having optical properties (florescence,brightness, color, etc.) indicative of calculus and/or plaque. In somevariations, this emphasized signal may result from the spectralillumination that creates no reflection in visible light, but creates asignificant fluorescence signal from plaque and calculus. For example,typical RGB illumination (using a common RGB sensor), may be modified toprovide amplification of the fluorescence signal (e.g., in near-IRregions) on the outer surface of the teeth. This amplification can beachieved by, as a non-limiting example, a large aperture that enables IRsignals to pass, and small aperture that enables the regular RGB(visible) spectrum to pass. This combination may produce color imageswith extra emphasis on fluorescent surfaces. Such fluorescence maymanifest in characteristic colors and brightness of the desired regionsindicating calculus and/or plaque on the teeth.

In any of the method and apparatuses described in which RGB images maybe take that include florescence signals (e.g., at a wavelength in whichplaque or calculus fluoresces), segmentation of the fluorescent regionsmay be performed on the image. For example, using the camera positionsduring acquisition of RGB and 3D scans (e.g., from the intraoralscanner), the fluorescent region may be registered with the 3D model(including the volumetric and/or just surface model) of the patient'steeth. This may result in a definition of relevant plaque and calculusregions on the final 3D model, which may further allow for definition ofthese regions, such as the borders of the calculus on the teeth, as wellas 3D surface & thickness of the plaque.

As already discussed above, regions on the 3D model may be compared withprevious/future scans of the same patient, which may show thedevelopment of calculus over time, and the effect of the calculus on thepatient's teeth. The apparatus may automatically or semi-automaticallymark (e.g., flag) these regions for monitoring. Thus, the size and shapeof calculus for each tooth may be monitored. Alternatively oradditionally, the thickness/depth of calculus may be compared withprevious scans. Any of this information may be provided quantitativelyand/or qualitatively, as discussed above. The thickness/depth ofcalculus may be compared with previous scans of clean teeth (includingone or more earlier scans following cleaning by a dental professional).This may provide an estimate of the thickness of the calculus in laterscans. As mentioned, measurements of the changes in the plaque, andparticularly the calculus over time may be made, and this data may beused to monitor plaque and calculus progression on the patient's teeth,and may provide well as visualization of the development.

In general, the monitoring and visualization of the patient's teethusing the methods and apparatuses described herein may be used as partof a dental and/or orthodontic treatment planning. As already mentionedabove, monitoring of calculus and plaque may be used to treatmentsincluding teeth cleaning. Scans may be performed prior to cleaning,during cleaning and/or after cleaning to provide guidance to the dentalpractitioner as to what regions to emphasize, focus on, or return to.Other treatments (coatings, caps, etc.) may be proposed based on theprogression of plaque and/or calculus over time. Further, monitoring ofany other feature or region of interest, including, e.g., caries,cracks, etc., as described above, may also provide treatment planninginformation. As discussed above, information about cracks and/or cariesmay be used to suggest treatments including restorations beforepotential issues develop further. In some variations, a digital model(e.g., surface and/or volumetric model) of the teeth may be modifiedusing the volumetric information, and the modified model(s) used todesign an orthodontic appliance or treatment plan. For example, a usermay digitally remove plaque and/or calculus from a volumetric scan takenprior or during a treatment. The modified scan may be used to guidetreatment, including further cleaning of the teeth, as necessary and toform or modify an appliance, so that the appliance (e.g., a dentalaligner) may fit better.

Combination with Dental Tools

The intraoral scanners and volumetric modeling described herein may beused and/or combined with other dental tools (drills, probes, etc.). Thecombined tool may provide numerous advantages.

For example, described herein are drills that may be used in conjunctionor combined with the intraoral scanners, and the use of 3D volumetricmodels. In some variations, a dental drill and an intraoral scanner maybe combined; e.g., incorporating a laser drill or laser-acceleratedwater drill into an intraoral scanner. This combination may allow thedental professional using the tool to directly visualize the tooth asand before it is drilled, providing real-time feedback to the user. Inone example, near-IR light may be applied to the probe head of the drill(e.g., laser drill) to provide imaging into the tooth, which will allowdirect forward-looking imaging prior and/or during drilling. The enameland dentin in the direct path of the drill may be imaged. The densityinformation can be used to inform the clinician when they have reachedthe dentin layer of a tooth or a certain depth inside the dentin, orwhen diseased regions have been removed. For example, the densityinformation can be used to provide haptic feedback to the operator,since tactile feedback is much more limited when using a dental laserversus a traditional headpiece.

The methods and apparatuses including intraoral scanners and volumetricmodeling as described herein may also be integrated into computer-aideddesign/computer-aided manufacturing technology for dentistry, asdescribed in FIG. 36. For example, dental implants, such as crowns(e.g., ceramic crowns) may be fabricated for an individual patient usingcomputer-aided design and computer-aided manufacturing (CAD/CAM)apparatuses and procedures. For example, traditionally CAD/CAMlaboratory manufacturing (“current workflow” in FIG. 36) may includepre-treatment scanning of the patient's teeth 3601 or an impression ofthe patient's teeth, such as a caries-free scan of the jaws. The teethmay then be prepared for the crown 3603, and then re-scanned 3605 andevaluated 3607. Finally, the crown may be made using CAD/CAM. Thesoftware for the CAD/CAM may receive the scanned information from thescanner and may process it for use in forming the design and performingthe manufacture. The use of CAD/CAM software may provide restorationscomparable to conventional restorations in all aspects, including inaesthetics, however the current methodologies may require repeated stepsfor evaluating and preparing the tooth, as shown by FIG. 36, andtypically require the user to perform these steps manually.

As shown in the “new workflow” on the bottom of FIG. 36, this method mayintegrate the 3D volumetric modeling described herein to simplify andimprove CAD/CAM of a patient's teeth. For example, the preparation maybe digitally designed, and this process may be automated (fully orsemi-fully, so that the user may approve and/or modify the process). Forexample, in FIG. 36, the pre-treatment scan 3611 may be performed usingan intraoral scanner that directly communicates with the CAD/CAMapparatus, or the intraoral scanner may include CAD/CAM capabilities. Inthis example, the tooth preparation 3613 may be fully digitally designedbased on the scan performed, and the scanner may guide the preparationof the tooth 3615. This may be done in real time with direct feedbackand/or guidance from the apparatus, which may integrate the scanner. Thescanner may then be used to evaluate the preparation 3617, on in somecases this step may be integrated fully into the guided prep step 3615,therefore removing the need for the post-prep evaluation. Finally,CAD/CAM may be used to prepare the crown (or other dental appliance) forthe correctly prepared tooth 3619.

Root Canal

The methods and apparatuses for 3D volumetric modeling of the patient'soral cavity (e.g., 3D volumetric modeling of the teeth) may also be usedto modify a root canal procedure. Typically, root canal proceduresrequire numerous x-rays to provide images into the teeth prior to,during, and/or after the procedure. The methods and apparatusesdescribed herein may remove or reduce the requirement for x-rays in thespecific example of root canal procedures. Specifically, as describedherein, an intraoral scanner including penetrative wavelengths (e.g.,near-IR) may be used to examine within a tooth, including within theroot of the tooth during the procedure. This may allow identificationand localization of the canal. For example, a tooth may be prepared forthe root canal by, for example, drilling a hole through the crown intothe tooth. The hole may be drilled with the guidance of the volumetricimaging described herein either during or interposed with the drilling.For example, a tooth (e.g., molar) may have an initial hole drilled intoit to expose the camber within the tooth. An intraoral scanner includingnear-IR may be used to image the tooth, including imaging through thehole that has been drilled into the tooth, to visualize into the pulpchamber. The scanner may be oriented, automatically or manually, toimage down into the chamber, which may allow visualization of the rootswithin the chamber. The initial drilling into the teeth may be limitedto penetrate the enamel and expose the inner chamber, and visualizinginto the chamber so that regions having different optical properties (atany wavelength, including in particular the near-IR wavelengths) maypenetrate into the chamber despite calcifications and/or infection, toallow imaging of the roots from within the tooth itself. The nervechambers of the root may be identified as being more or less dense thanthe surrounding regions within the dentin and enamel. By removing theroof of the chamber to expose the inner pulp region of the tooth, theintraoral scanner may visualize through the drilled opening to provideadditional volumetric information, including the locations, curvature,and trajectory of the tooth root. Detection of hidden canals andaccessory canals may be facilitated by this additional visualizationinformation. This information may then be used to guide the therapy.

For example, in some variations, the method may include taking, using anintraoral scanner as described herein, a 3D volumetric model of thepatient's teeth either before or after drilling to form an opening intothe target tooth (e.g., for which a root canal will be performed). Thedrilling may be done with or without guidance from an intraoral scanner,as described above. The inner chamber of the tooth may be visualizedusing the intraoral scanner, e.g., through an opening drilled from thecrown of the tooth. The apparatus may then determine the locations ofthe horns of the pulp chamber for the tooth. Any of the methodsdescribed herein may be used in combination with x-ray information.Treatment planning may be performed by the apparatus to determine theshape and/or location of the pulp horns, pulp chamber, and roots to mapout a treatment plan for drilling/tissue removal that avoidsoverthinking or breaching the lateral sides of the tooth. This treatmentplan may then be used to guide the user in drilling on the teeth, and/orfor automating the drilling. In some variations the drill may bedirectly guided by imaging, e.g., using the hybrid drill/intraoralscanner described above. Alternatively or additionally, roboticassistance may be provided using the treatment plan. In some variations,the procedure may be performed manually, and the drilling may be done insmall increments, with visualization between drilling steps to confirmthe treatment path, and avoid over-drilling, as well as confirming thatthe entire region has been drilled and infected pulp removed. Additionalvisualization (including using a contrasting agent) may be used.

In general, any of the methods described herein, including the rootcanal methods described above, may be used with one or more contrastingagents during imaging. For example, contrasting agent may includematerial applied to the outside of the tooth (or into a hole or openingin the tooth, including holes drilled into the tooth). Contrast agentsthat absorb or reflect in the near-IR, or other wavelengths use by theintraoral scanner may be used. Preferably contrast agents may be usedthat are distinguishable at some of the imaged wavelengths, but not allof them, to provide differential imaging. For example, contrast agentsmay be visualizable under white light, but not near-IR; alternatively, acontrast agent may be visualizable under near-IR but not white light, orunder some wavelengths of near-IR but not others for which images aretaken. Contrast agents that preferably attach mix or coat with one ormore targets within the teeth or oral cavity may be used. For example, acontrast agent that selectively binds to one or more of: bacteria,plaque, calculus, gingiva, pulp, etc., may be used. In use, the contrastagent may be applied to the teeth/oral cavity, rinsed, then visualized(or visualized without rinsing). For example, a contrast agent thatabsorbs IR light may be used for inclusion as part of, or mixed in witha material forming, e.g., a dental implant (e.g., to fill a cavity, capa tooth, fill a root canal, etc.,) to create an IR contrasting fillermaterial that may be easily visualized when scanning as describedherein.

Also described herein are methods of determining improvements in softtissue around the teeth using the apparatuses and methods for generating3D volumetric models of the teeth, as described herein. For example agum recession may be monitored and/or quantified, and may be observedover time using these methods and apparatuses. In addition to the directvisualization of plaque and/or calculus as described above, the methodsand apparatuses described herein may also or alternatively detect theeffect on the teeth, including recession of the bone due to plaque andcalculus. Diseased regions, may be visualized directly. In somevariations a contrast agent may be used to provide additional contrastfor the intraoral scanner to detect diseased regions. Scanning of thesurface of the gingiva may identify inflamed and/or discolored regionsthat may be indicative of gum disease. This information may be combinedwith the 3D volumetric modeling of the teeth, including the location ofplaque and/or calculus, as discussed above.

FIGS. 37A and 37B illustrate an example of a monitoring, over time, gum(gingival) recession. In this example, the display may show the 3D modelof the teeth and a comparison between the original scan, and asubsequent scan, taken 2-3 years later. In FIG. 37A, the two scans havebeen aligned and compared, and differences shown by a color indicator,e.g., a heat map. In FIG. 8, darker colors (which may be shown in color,e.g., red) show recession of the gingiva to a greater degree. Thecircled region B in FIG. 37A is shown in greater detail in FIG. 37B forthe later scan. Although FIG. 8 illustrates primarily surface features(e.g., gingiva position), volumetric information may be used to generatethis information, e.g., showing changes in gingiva thickness and/orvascularization, enamel thickness, etc.

In addition to guiding the user and/or dental technician based on thescans (e.g., showing plaque, calculus and/or inflammation inparticular), these methods and apparatuses may be used by the dentalprofessional to rate, rank or quantify the removal of plaque and/orcalculus, either immediately following a treatment, or over time. Thismay provide a metric against which treatments may be judged. The scaninformation may also be used to provide information to the patients,including a map or guide for home treatment, including which areas tofocus on brushing, flossing, etc. The guide may include one or moreimages from the 3D volumetric model, for example. Guidance informationabout what teeth or oral cavity regions to focus home dental care (e.g.,brushing) on may be provided to an electronic toothbrush that may alsohelp guide the patient in brushing based on identified regions.

The methods and apparatuses described herein may also be used withpatient's already having a dental appliance installed on the teeth,including braces, bridges, and the like. For example, in some variationsthe patient may include 3D representations of the dental appliance, andmay provide information to help design or modify future dental devices(e.g., retainers, aligners, braces, etc.).

In particular, the methods and apparatuses described herein may be useto provide very accurate volumetric and surface information about thepatient's teeth that may be useful for treatment planning of any type ofdental treatment. In some variations the methods and apparatusesdescribed herein may be useful for treatment planning of an appliance,such as an aligner or retainer, that is optimally worn in closeproximity to the patient's teeth. For example, a method and/or apparatusthat includes a 3D volumetric scan of the patient's teeth may be used tosubtract out or remove from the 3D model of the teeth, any plaque,calculus and/or food debris that might be present at the time of the 3Dscan. By digitally subtracting out any plaque, calculus, and/or fooddebris present, the volumetric information may be used with a virtualrepresentation of an aligner, retainer, night guard, or other device,and the fit improved prior to fabricating, applying or wearing theapparatus.

The gingival tissue surrounding the teeth, being of different density(or different optical absorption/reflection properties) than the enamel,may also be identified and characterized with greater accuracy, so thatthe junction between the inner contour and the tooth surface can beidentified. By doing this, the shape of the tooth surface beneath thegingival tissue can be accurately characterized so that predictivemodels of tooth movement can have more accurate representations of theteeth as portions of the teeth not initially visible become graduallyexposed as the teeth align. In other words, some parts of the teeth maybe initially obscured by the gingival tissue, but as the teethstraighten, the gingival tissue migrates, and the previously-coveredregions are exposed. By detecting the tooth regions beneath the gingivaltissue accurately, the future state of the teeth after the gingiva hasmigrated can be more accurately modeled.

The methods and apparatuses described herein may also be sued to detect,diagnose, and/or treat disorders of the oral cavity.

For example, the 3D volumetric scanning and modeling methods andapparatuses described herein may be used to detect and/or treat salivarystones (e.g., plugging of the salivary ducts). These glands, which maybe located near the molars and under the patient's tongue, may bescanned using the intraoral scanners described herein. These scans maypenetrate the soft tissue and may detect the hard, stone-like formations(i.e., sialoliths, salivary-gland stones, or duct stones) that arecalcified structures that may form inside a salivary gland or duct andblock the flow of saliva into the mouth. The methods and apparatusesdescribed herein may be used to identify these structures and/or mayguide and/or confirm removal of these stones.

In addition to or instead of the use of the apparatuses and methodsdescried herein to identify, diagnose and/or track regions, includingpre-cavitation caries, crack, etc., the methods and apparatusesdescribed herein may also or alternatively be used to identify andmanipulate regions that have already been modified. For example,fillings, attachments (for attaching an angler, braces, etc.), braces,retainers, etc. and any other structures may be identified within thevolumetric model and/or displayed. For example regions of enamel and/orenamel-like restorations may be displayed differently in the volumetricmodel. These regions will typically have different optical properties,including different scattering/absorption of the near-IR (and in somecases visible light) compared to each other and/or other regions of theoral cavity, including the dentin. Such regions may be manually,automatically or semi-automatically identified, and may be segmentedand/or separately manipulated. For example, in some variations theseregions (e.g., attachments/cement, etc. for an aligner or otherappliance) on the tooth may be identified for removal by the dentalpractitioner, and the 3D volumetric model or data (images) taken from itmay be provided to guide such treatment. They may also or alternativelybe digitally subtracted to provide a better fit for a new appliance onceremoved. A subtracted view may also or alternatively be provided to apatient.

In some variation the internal structural integrity of an artificialdental structure or modification (e.g., dental bond, filling, etc.) maybe determined using the volumetric model(s) described herein. Forexample, a volumetric model may include internal detail of an artificialdental structure, such as the structural detail within a filling, bond,etc., or the interface between the natural tooth (enamel, dentin, etc.),and this information may be presented or shown to the user in detail toallow an assessment (or to allow automatic assessment) of the conditionof such artificial dental structures. This may facilitate their removal,repair and/or replacement.

The 3D volumetric models of the teeth (and method and apparatuses forgenerating them) may also be used as a diagnostic or detection tool forfuture tooth sensitivity. For example, an abfraction is a form ofnon-carious tooth tissue loss that typically occurs along the gingivalmargin. The abfraction lesion may be a mechanical loss of toothstructure that is not caused by tooth decay that may occur in both thedentin and enamel of the tooth. These are believed to be caused byrepetitive stress cycles from the patient's occlusion, and exacerbatedby aggressive brushing. The 3D volumetric models of the teeth enhancedby density analysis of the enamel and dentin near the gingival line mayprovide an early indicator of these lesions. For example, an apparatusmay examine the volumetric model to identify the initial stages offormation for these crescent-shaped lesions. Multiple 3D volumetricmodels taken over time may indicate the rate of progression of theselesions. A system may be configured to automatically or manuallyidentify them; as described above, they may be automatically orsemi-automatically flagged.

Thus, the apparatus and methods may identify and alter the user thatsuch a “hotspot” leading the future tooth sensitivity may be occurring,and may provide for treatment plans to slow, stop or reverse theprogression of the lesion. Tooth sensitivity can result from these smallfractures and the exposed dentin. Detection may be triggered byidentifying the characteristic crescent shape that develops in the moremature lesions, however earlier detection may be made by identifyingregions of thinning in the enamel and/or dentin (e.g., near the gingivalline), which may progress over time. The apparatuses and methods mayflag and/or assign risk based on the actual thickness and/or theprogression of changes in the thickness.

The methods and apparatuses described herein may also be used to detectthe development of acid reflux, based in part on characteristic wearpatterns, and/or changes (e.g., over time) in the enamel thickness ofthe patient. For example, acid reflex while a patient sleeps may resultin the gradual erosion of the patient's teeth in a characteristicpattern (e.g., from the back of the teeth, on the lingual side. Assimilar pattern may develop with bulimia. The volumetric models of thepatient's teeth taken, e.g., by near-IR, may provide an accurate mappingof the enamel density and thicknesses of all of the patient's teeth.Thus, a method of detecting acid reflux (or bulimia) may includedetecting (including detecting over time) characteristic thinning of theenamel of the patient's teeth in the rear, lingual region. The moreproximal, lingual region of the teeth may have an unusually thinner (orthinning) enamel thickness, compared to more anterior (forward) regionson the opposite, buccal, side of the patient's teeth.

The methods and apparatuses described herein may also be used to detectthin enamel regions from occlusal wear due to chronic grinding of thepatient's teeth and/or predict tooth sensitivity that may result fromthis grinding. 3D volumetric models of the patient's teeth may show asnapshot of the occlusal thickness of the patient's enamel and theproximity of the dentin to the occlusal surface. Further, multiple scanstaken over time may show the loss of enamel in the occlusal surface.This was mentioned above as one indicator that may be automatically,manually or semi-automatically marked or flagged. For example, a flagcan be set whenever a region >0.5 mm² develops within 0.5 mm of dentin,and the regions of the digital model highlighted. This allows anyregions which satisfy the flag criteria to be visualized and/ormonitored. Given a patient's age and in some variations gender, as wellas the changes in the enamel thickness over time, an estimate of thewear rate over time may be provided, along with proximity to dentinregions, and thus an estimate or prediction of the tooth sensitivity orpain may be made. Grinding of teeth may also be an indicator of otherissued, including sleep apnea. For example, sleep apnea may also bedetected from 3D volumetric models of the patient's teeth, particularlyover time. Many patients with sleep apnea grind their teeth (e.g., in aforward and/or side to side motion), which may result in a pattern oferosion of the teeth. Thus, the methods and apparatuses described hereinmay be used to help diagnose or confirm sleep apnea.

In general, any of the methods and apparatuses described herein may beused with non-human patients. For example, any of the methods andapparatuses described herein may be used as with veterinary patient's(e.g., animals) to determine, for example, the state of the animalsteeth, including wear on the teeth.

The methods and apparatuses described herein may also be used to providean estimate of risk for the patient in developing fractures in theteeth, and/or the development of tooth sensitivity. For example, a the3D volumetric models of the teeth described herein may be used toidentify malocclusions in the teeth and resulting wear and/or crackingof the teeth, based on the mechanical estimates of the tooth thicknessand wear pattern. Functional information such as chewing pattern andarticulation forces may also be integrated into the assessment. Wearpatterns may be identified and shown as ‘hotspots’ for example on imagesgenerated from the 3D representation of the patient's teeth. This may bedisplayed to the patient as information, including as informationwarning of potential risks. High risk regions may be identified to thepatient along with an explanation of the potential risk.

In general, the methods and apparatuses, and particularly the monitoringand comparison, over time, of 3D volumetric models including informationabout the internal structures of the teeth (e.g., enamel and dentindistribution within the teeth) may be used to identify, monitor,diagnose, and guide treatment of a variety of disorders in addition tothose mentioned above. For example, dentin dysplasia, enamel dysplasia,etc. These methods also allow the identification of multiple differenttypes of enamel within the patient's teeth, including regions havingdifferent amounts hydroxyapatite, amelogenins and/or enamelins, ordifferently organized regions of these, including regions that arehomogenous or non-homogeneous, and that may have different opticalproperties for the near-IR wavelengths used for imaging.

Interactive Display of 3D Model of a Patient's Dental Arch

As already described (and shown in the figures above), the methods andapparatuses described herein may allow a user to virtually scan apatient's dental arch. In particular a 3D model of the patient's dentalarch(s), which may be volumetric, surface, or both (or in somevariations an abstracted or generic model), may be used in conjunctionwith images taken, e.g., using an intraoral scanner, from variouspositions around the dental arch. These images may be images that wereused to generate the 3D model of the dental arch. The images may betagged and/or arranged in the data structure to indicate theircorresponding position or region or angle relative to the 3D dental archmodel. In some variations, the 3D model and the images taken may bemaintained as a data structure, however it is not necessary that the 3Dmodel be included with the images as a single data structure.

For example, FIG. 42 is an example of a data structure that includes oneor more dental arch models 4205 as well as a plurality (e.g., greaterthan 50, greater than 100, greater than 200, greater than 500, greaterthan 750, greater than 1000, greater than 10,000, etc.) of one or more(e.g., sets) of images taken from positions around the patient's dentalarch. In some variations both visible light and near-IR (or near-IR andother modalities) images 4201 may be shown and may share positionalinformation. The positional information typically includes the region ofthe dental arch (e.g., in x, y, z coordinates, such as the coordinate ofa center point of the image relative to the dental arch) from which theimage was taken, as well as the angle (e.g., roll, pitch and/or yaw, orradial coordinates, etc.) relative to the plane of the dental arch(“positional info” 4201). In some variations the scans may be compositesof multiple scans (e.g., averages, blends, etc.) that are combined andstored in the data structure. The 3D model may be formed by virtually“stitching” the scans together to form the 3D model.

The data structure may be stored in a compressed configuration; althoughit may contain a large amount of data, the compression and organizationof the data structure may allow it to be manipulated for display. Forexample, FIG. 41 illustrates one method of interactively displaying a 3Dmodel of a patient's dental arch using a data structure such as the oneschematically shown in FIG. 42.

In FIG. 41, the method includes displaying the 3D model of the patient'sdental arch 4101 and displaying on the 3D model a viewing window 4103.The user may then be allowed to continuously move the two (e.g., eitheror both the viewing window and the 3D model) so that the teeth of thedental arch may be virtually viewed “though” the viewing window ingreater detail in a nearby view 4105. The angle of the viewing window aswell as the location of the viewing window along the dental arch may bechanged by the user, e.g., moving continuously over and/or around the 3Dmodel of the dental arch 4107. As the viewing window/dental arch aremoved relative to each other, a corresponding image (or images), such asa near-IR image, taken at a position relative to the dental archcorresponding to the position of the viewing window, may be identifiedfrom the data structure/data set (e.g., FIG. 42) 4109. The correspondingimage(s) may then be displayed 4111, and this process may be iterativelyrepeated as the viewing window is moved over and along the 3D dentalarch model.

In some variations, the data structure may be configured or arrangedtopographically or in an indexed topographic manner; thus images ofadjacent regions may be linked or ordered in the data structure,simplifying the method.

FIGS. 43A-45C illustrate examples of one variation of a user interfacethat may allow the user to virtually scan a 3D model of the dental arch,showing corresponding images (e.g., near-IR images) as described in FIG.41. As mentioned above, the near-IR images may be viewed by the user toidentify manually (or in some variations automatically) identify one ormore structures/defects and/or actionable dental features, includingdental caries, cracks, wear, etc. The display of corresponding 3D dentalarch model and visible light images of the same regions may both giveperspective and allow for immediate comparison with the patient's teeth,simplifying and powerfully augmenting dental analysis.

For example, in FIG. 43A, the display is shown as a user interface 4300including a dental arch model 4303 (3D dental arch model) reconstructedform scans of the patient's teeth and stored, along with many or all ofthese scans, in a data structure. As already mentioned above, it is notnecessary, but may be helpful, for the 3D dental arch model to beincluded in the data structure with the plurality of images. Further,the 3D dental arch model in this example is constructed from the scansof the patient's teeth, however, is should be clear that the 3D dentalarch model may be non-representative, and yet may be used to select the2D views to be displayed, as described herein. A viewing window 4301,shown as a loop or circle, may be moved over or along the 3D model ofthe dental arch; as the viewing window is moved, each of two imagedisplays 4305, 4307 are updated with images corresponding to theposition (both the region of the dental arch and the angle of the dentalarch relative to the plane of the viewing window. In FIG. 43A the first(upper) image 4305 is a near-IR image and a corresponding (taken at thesame approximate time/location) visible light (e.g., color) image isshown in the bottom image 4307. Alternatively displays are shown inFIGS. 43B and 43C, showing just a single image each; in FIG. 43B anenlarged near-IR display image is shown, while in FIG. 43C a single,enlarged visible light display image of the region corresponding to theimaging window view is shown.

The user interface may also include tools 4309 for manipulating thedisplay (e.g., rotating, moving the dental arch and/or viewing window,modifying, marking, etc., the images and/or 3D model, saving,opening/recalling images, etc.

FIGS. 44A-44B illustrate an example of moving the viewing window overthe teeth and changing/updating the corresponding images. FIG. 44A showsthe image of the dental arch with corresponding near-IR and light imagesas “seen” through the viewing window at a middle region of the dentalarch. In FIG. 44B the dental arch has been rotated by the user (oralternatively, the viewing window has been rotated relative to thedental arch lingually) so that the viewing window is slightly linguallypositioned relative to FIG. 44A; the corresponding views (near IR andvisible light) have been updated in real time to show this change of therelative position of the viewing window.

Similarly, FIGS. 45A-45C shows an example of a 3D model of a patient'slower arch similar to the view shown in FIG. 43A-43C. In use, as theuser scans over and along the dental arch by moving the viewing window(and/or the dental arch relative to the viewing window), the displayimages may change virtually continuously, so that they may update inreal or near-real time. The user may identify features in the near-IRimage(s), including densities changes in the region of normallyIR-transparent enamel, which may indicate carries, cracks, or wearing inthe enamel.

The intraoral scanning system shown in FIGS. 1A-1B may be configured asan intraoral scanning system. Returning to FIG. 1A, the intraoralscanning system 101 includes a hand-held wand 103 having at least oneimage sensor and a light source configured to emit light at a spectralrange within near-infrared (near-IR) range of wavelengths, and a displayoutput (screen 102). The screen may be a touchscreen acting as a userinput device, or the system may include a separate user input device(e.g., keyboard, touchpad, joystick, mouse, track ball, etc.). Asindicated in FIG. 1B, the system may also include one or more processorsthat are operably connected to the hand-held wand, display and userinput device. The one or more processors may include circuitry and/orsoftware and/or firmware configured to: display a three-dimensional (3D)model of a patient's dental arch on the display output; display aviewing window over a portion of the 3D model of the patient's dentalarch on the display output; change a relative position between theviewing window and the 3D model of the patient's dental arch based oninput from the user input device; identify, from both the 3D model ofthe patient's dental arch and a plurality of images of the patient'sdental arch taken from different angles and positions relative to thepatient's dental arch, a near-infrared (near-IR) image taken at an angleand position that approximates a relative angle and position between theviewing window relative and the 3D model of the patient's dental arch;and display the identified near-IR image taken at the angle and positionthat approximates the angle and position between the viewing windowrelative to the 3D model of the patient's dental arch (as shown in FIGS.43A-45C).

Automatic Characterization of Dental Features

Also described herein are methods and apparatuses (e.g., systems,including software) that is configured to use the 3D models, includingbut not limited to the volumetric 3D models, of all or a portion of apatient's dental arch to automatically or semi-automatically identify,confirm and/or characterize one or more dental feature. In particular,these methods and apparatuses may be configured to identify, confirm,and/or characterize one or more actionable dental features that maybenefit from detection and/or treatment. Actionable dental features mayinclude, but are not limited to cracks, gum recess, tartar, hard tissueand soft tissue oral conditions, etc. Enamel thickness may be anotheractionable dental feature. For example, the methods and apparatusesdescribed herein may automatically map enamel thickness (e.g., applycolor map where enamel is lower than x microns thick, where x may bepreset and/or user adjustable). Areas of thin enamel are potential areaswhere caries may exist. Other potential actionable dental features mayinclude discoloration (e.g., discontinuities in color), pits, fissures,evidence of grinding (thinning, including thinning over time),interproximal voids, etc., or any other similar feature that may beindicate or suggestive of where caries are likely to form.

Any of the methods and apparatuses described herein may use multipledifferent images or sets of images of the patient's teeth taken withdifferent imaging modalities are used to detect, analyze and/orcharacterize dental features, and particularly actionable dentalfeatures. The multiple different images or sets of images of thepatient's teeth taken with different imaging modalities may each bereferred to as a “record”. Each record may be a different imagingmodality, such as dental cone beam computed tomography (CBCT) scanning,three dimensional (3D) intra-oral scanning, color scanning (one or moreof: 3D color scanning, surface color scanning, etc.), two-dimensional(2D) color scanning, near-IR scanning (including, but not limited to oneor more of: volumetric near-IR imaging, trans illumination and/orreflective scanning), X-ray (including, but not limited to:cephalometric analysis x-ray scanning, panoramic x-ray scanning, etc.),etc., and may include text or graphic chart information of the patient.

For example, each record may initially be processed independently. Oneor more dental features, and in particular, one or more actionabledental features, may be identified by this initial scan. A single record(e.g., a single imaging modality) may be used first to identify the oneor more actionable dental features, or all of the records, or a subsetof the records may be initially processed to identify the one or moreactionable dental features. The initial identification of the one ormore actionable dental features may be performed manually orautomatically or semi-manually. For example, one or more actionabledental features may be identified automatically; a system as describedherein may review the record (including the one or more images of thepatient's teeth) to flag or identify regions having a characteristicassociated with an actionable dental feature. A system may be trained,using machine learning techniques such as supervised learning techniques(e.g., classification, regression, similarity, etc.), unsupervisedlearning techniques (e.g., density estimation, cluster analysis, etc.),reinforcement learning (e.g., Markov Decision Process techniques, etc.),representation learning techniques and/or principle component analysis,etc., to identify/flag a region of a particular scan in a specifiedmodality that is associated (even loosely associated with) an actionabledental characteristic. Alternatively or additionally, a user (dentalprofessional, technician, etc.) may manually review one or more records(each in a particular imaging modality) and may flag or identify regionssuspected to show an actionable dental characteristic. In asemi-automated configuration the system may initially flag one or moreregions from a record that the user may then review and confirm/reject.

As the one or more regions are identified, they may be flagged and/orstored in a collection of potential actionable dental features. Thelocation may be relative to (e.g., the location on) the originatingrecord, or relative to a reference model (such as the 3D volumetricmodel, as will be described in greater detail below). In some variationsthe collection (e.g., array, data structure, file, etc.) may alsoinclude one or more of the type of potential actionable dental features,the extent of the potential actionable dental features, a grade and/ordegree of the potential actionable dental features, the originatingrecord and/or the imaging modality of the originating record, etc. Insome variations the data structure may be integrated into theoriginating record (or a copy thereof) and may modify the image(s) ofthe originating record, e.g., by include a flag or marker at thelocation of the identified potential actionable dental features and/orany meta text such as the grade and/or degree, etc. The grade and/ordegree may refer to the confidence level or score for the potentialactionable dental feature, including the confidence level or score thatthe identified potential actionable dental features is likely ‘real’.

This initial identification process to identify potential actionabledental features may be performed across multiple records, or it may belimited to a subset of the records (e.g., including just to one of therecords), as mentioned above. In some variations the process may beiteratively performed.

Once one or more potential actionable dental features is identified, itmay be cross-referenced to the other one or more records that use(s)other imaging modalities. Thus, the locations of the one or morepotential actionable dental features may be examined in particulardetail to determine if the same potential actionable dental feature isapparent on these one or more other record. In some variations theentire additional record(s) may be examined during this confirmationportion of the procedure, and any additional potential actionable dentalfeatures from the additional one or more records may be likewise flaggedas a potential actionable dental feature and the same region of thedental arch may be examined for these other potential actionable dentalfeatures (including returning back to records that have already beenreviewed, such as the first or originating record).

Comparison across other records may be guided by translating thelocations of the dental features (including but not limited to thepotential actionable dental features) between the different records. Inparticular, it may be helpful to coordinate the individual dentalrecord(s) begin examined to a model of the patient's dental arch, suchas any of the 3D models, and in particular, the 3D volumetric models,described above. The 3D model of the dental arch may therefore act as akey to translate the locations of the one or more potential actionabledental features and may allow rapid and efficient comparisons betweenthe different records, e.g., different imaging modalities.

Thus, a correlation between each of the different records and, inparticular, a correlation between all or some of the different recordsand a 3D model (e.g., a 3D volumetric model) of the dental arch may beestablished either before or after the initial scan for potentialactionable dental features. Any method of correlating a records andother records and/or a 3D model of the patient's dental arch (or aportion of the dental arch) may be used. For example, one or more easilyrecognizable features (e.g., tooth edge, shape, segmentation, etc.) maybe used to determine landmarks that may translate between the one ormore records and/or the one or more records and the 3D model of thepatient's dental arch. In some variations a translational dataset may becreated that includes a transformation between the records and/orbetween each record and a 3D volumetric model of the patient's dentalarch. For example, a 3D volumetric model of all or a portion of thedental arch may include transformation information for each of the oneor more records allowing transformation of the image(s) of the one ormore records, such as an estimate of the distance and/or orientation ofthe imaging modality relative to the record image(s). This allows bothforward and reverse translation of position between each record and the3D model (e.g. volumetric model).

Thus, a translational dataset may include a 3D model and thetranslational information of each record, so that a portion or region ofa record image (or images) may be projected onto the 3D (translational)model, and the same region then back projected onto a second (or more)record taken with another imaging modality so that the same region maybe examined. In some variations the process may begin with thecollection of all the records and/or an automatic, manual orsemi-automatic registration between all the records. For example, theidentification of individual teeth, palate, gingiva, etc. regions, maybe used to cross-correlate between the different imaging modalitiesand/or the 3D model. In one example, a record including x-ray images maybe correlated with a 3D volumetric model of the patient's teeth bysolving (manually or automatically) for the position and/or orientationof the x-ray camera taking the x-ray images corresponding to the record.The volumetric model may be used to determine and/or confirm thelocation and/or orientation of imaging source for each record. In somevariations the record include explicit (e.g., recorded) informationabout the position and/or orientation and/or imaging parameters used totake the image(s); alternatively or additionally, this information maybe derived. As described above one a pseudo-x-ray image may be generatedand compared to an actual x-ray image of the record.

Once a region corresponding to the region of the potential actionabledental feature from another record is identified, the system or methodmay then determine if the same potential actionable dental features ispresent in this other record. If present, the score (e.g. confidencescore, showing the likelihood that the potential actionable dentalfeatures is real) may be adjusted, e.g., increased if the same or asimilar potential actionable dental feature is present. Depending on thetype of record and the type of potential actionable dental features, theabsence of a potential actionable dental features may result inadjusting the confidence score. For example, the absence of surfacefeatures that are not typically detectable by X-rays, such asdiscoloration, plaque, gum recession, etc., may not result in loweringthe confidence score of the one or more potential actionable dentalfeatures. The more occurrences of finding a potential actionable dentalfeatures a corresponding location between different records (thereforein different modalities), the more likely that the potential actionabledental features really exists.

In comparing the corresponding locations of the one or more potentialactionable dental features the region may be examined manually,automatically or semi-automatically, similar to the originalidentification techniques discussed above. For example, a region of anadditional record corresponding to the location of a potentialactionable dental feature in another record may be examinedautomatically to identify features correlated with the type of potentialactionable dental feature. The system may be trained to recognize thepotential actionable dental feature in the imaging modality of theadditional record and may provide a score indicating the likelihood thatthe potential actionable dental feature is present in this location. Insome variations a user (e.g., technician, dental professional, etc.) maybe presented with an image from the additional record(s) and maymanually indicate the likelihood (yes/no, graded scale, numeric scale,etc.) that the potential actionable dental feature is present in the oneor more additional records.

The final confidence value determined for each potential actionabledental feature may be used by the system: stored, transmitted and/ordisplayed. For example the potential actionable dental feature(s) may bepresented to a dental practitioner in any appropriate manner, includingin a list, on a display, such as on 3D model of the dental arch(including the translational 3D dental model) marked, etc. For example,the system may output a display highlights by color, shape, etc. thelocation of any or all of the potential actionable dental features thatare above a threshold confidence level (so likely to be ‘real’); thedisplay may also include one or more views (from the one or morerecords) of the potential actionable dental feature. The user may set ofadjust the threshold confidence level, including on the fly (e.g.,making the threshold more or less stringent and showing the addition orremoval of potential actionable dental features in response.

FIG. 46 illustrates one example of a method 4600 for characterizingdental features across different imaging modalities as just discussed.In FIG. 46, the method (or a system configured to perform it) mayidentify one or more actionable dental features from one or more records(e.g., one or more images or sets of images of the patient's teeth takenwith different imaging modalities) 4601. For example, the one or moreactionable dental features may be identified by an agent or engine thatis configured to automatically detect one or more actionable dentalfeatures. For example, a system performing the method of FIG. 46 mayinclude an actionable dental feature analysis engine, or may includemultiple actionable dental feature analysis engines each configured toidentify one or more types of actionable dental features or one or moretypes of imaging modality. The engine (e.g., an actionable dentalfeature analysis engine) may be part of a computer system. As usedherein, an engine includes one or more processors or a portion thereof.A portion of one or more processors can include some portion of hardwareless than all of the hardware comprising any given one or moreprocessors, such as a subset of registers, the portion of the processordedicated to one or more threads of a multi-threaded processor, a timeslice during which the processor is wholly or partially dedicated tocarrying out part of the engine's functionality, or the like. As such, afirst engine and a second engine can have one or more dedicatedprocessors or a first engine and a second engine can share one or moreprocessors with one another or other engines. Depending uponimplementation-specific or other considerations, an engine can becentralized or its functionality distributed. An engine can includehardware, firmware, or software embodied in a computer-readable mediumfor execution by the processor. The processor transforms data into newdata using implemented data structures and methods, such as is describedwith reference to the figures herein.

The engines described herein, or the engines through which the systemsand devices described herein can be implemented, can be cloud-basedengines. As used herein, a cloud-based engine is an engine that can runapplications and/or functionalities using a cloud-based computingsystem. All or portions of the applications and/or functionalities canbe distributed across multiple computing devices, and need not berestricted to only one computing device. In some embodiments, thecloud-based engines can execute functionalities and/or modules that endusers access through a web browser or container application withouthaving the functionalities and/or modules installed locally on theend-users' computing devices.

Returning to FIG. 46, the one or more actionable dental features may beidentified from one or more records manually or semi-manually. Forexample, an actionable dental feature analysis engine may initiallyidentify one or more actionable dental features that may then beverified or vetted by a user (e.g., dental technician).

Each actionable dental feature identified may then be flagged and/orrecorded, e.g., in a collection of potential actionable dental features4603. For example a collection of potential actionable dental featuresmay be part of a data structure. Adding the potential actionable dentalfeature(s) to a collection (e.g., data structure) may include recordinga location of the actionable dental feature (e.g., on the originatingrecord) and/or one or more of: type of actionable dental features,grade/degree of confidence of the actionable dental feature, etc. Asused herein, a data structure (which may be included as part of adatastore) is intended to include repositories having any applicableorganization of data, including tables, comma-separated values (CSV)files, traditional databases (e.g., SQL), or other applicable known orconvenient organizational formats. Datastores can be implemented, forexample, as software embodied in a physical computer-readable medium ona specific-purpose machine, in firmware, in hardware, in a combinationthereof, or in an applicable known or convenient device or system.Datastore-associated components, such as database interfaces, can beconsidered “part of” a datastore, part of some other system component,or a combination thereof, though the physical location and othercharacteristics of datastore-associated components is not critical foran understanding of the techniques described herein.

A data structure may be associated with a particular way of storing andorganizing data in a computer so that it can be used efficiently withina given context. Data structures are generally based on the ability of acomputer to fetch and store data at any place in its memory, specifiedby an address, a bit string that can be itself stored in memory andmanipulated by the program. Thus, some data structures are based oncomputing the addresses of data items with arithmetic operations; whileother data structures are based on storing addresses of data itemswithin the structure itself. Many data structures use both principles,sometimes combined in non-trivial ways. The implementation of a datastructure usually entails writing a set of procedures that create andmanipulate instances of that structure. The datastores, describedherein, can be cloud-based datastores. A cloud-based datastore is adatastore that is compatible with cloud-based computing systems andengines.

The identified “putative” actionable dental features (e.g., “potentialactionable dental features”) may be mapped to corresponding physicallocations in one or more other records 4605. As discussed above, in somevariations this may be done using the 3D volumetric model, which maytranslate between the various different types of records (havingdifferent imaging modalities), including projecting a first record ontothe 3D model and then back onto a second region.

Thus, the same corresponding regions in other records may be reviewed todetermine if the potential actionable dental feature is present orsuggested in the additional record(s). In some variations, the methodmay simply collect all of the different corresponding regions forstorage, transmission and/or presentation to a user (e.g., dentalprofessional), e.g., optionally stopping here and allowing the user toreview these flagged region from multiple different imaging modalities(records) in parallel. For example, the potential actionable dentalfeature may be shown for all corresponding views in a side-by-side(e.g., tiled) or sequential view(s).

Alternatively or additionally, the method and/or system mayautomatically or semi-automatically adjust a confidence score for eachof the potential actionable dental features identified. Thus, the systemmay determine if the additional records indicate that the potentialactionable dental feature is more likely to be present or less likely tobe present and may adjust (or determine) the confidence score for eachof the potential actionable dental features, based on the appearance atthe corresponding location in the additional record(s) 4607.

The adjusted confidence levels may then be used to narrow down thepotential actionable dental features. For example, the method or systemmay then filter and/or apply a threshold based on the adjustedconfidence level for each potential actionable dental feature 4609. Insome variation the threshold may be fixed (e.g., confidence level ofgreater than x, where x is a numeric value intermediate between zeroconfidence and 1 (absolute confidence), e.g., 0.1, 0.2, 0.3, 0.4, 0.5,0.6, 0.7, 0.8, etc. In some variations the threshold value may bemanually adjusted by the user and/or may be based on one or morefeatures of the records, such as a quality metric specific to each ofthe records, etc.

Potential actionable dental feature having a confidence level that isabove the threshold value may then be stored, presented and/ortransmitted 4611. For example a user may be presented with a final listand/or display (e.g., using the 3D model) including the flaggedpotential actionable dental features.

Any of the methods and apparatuses (e.g., systems) described herein maybe configured to build a data structure including all or part of themultiple records. For example, a data structure may include a 3Dvolumetric model and all or some of the associated 2D images that wereused to construct it, as described above. In addition, the datastructure may include additional records, such as images taken by X-ray(e.g., panoramic), and/or CBCT, etc. Metadata, e.g., information,including textual information, about the patient and/or images may alsobe included, including optionally patient chart information from thepatient's health/dental records. Alternatively or additionally, anyidentified potential actionable dental features (e.g., findingsidentified from the records) may also be included. The potentialactionable dental features may be used to search/find/mark on the otherrecords.

Typically, when compiling the images (e.g., 2D near-IR images) to buildthe 3D (e.g., volumetric) model, the 2D images that provided informationmay be marked to indicate the significance to the 3D model. For example,2D images may be marked as less relevant or more relevant.

As mentioned above, the collection of potential actionable dentalfeatures, including their confidence level based on their presence inmultiple records may be included as part of the same data structureincluding the 3D model, or it may be separate. The 3D model may bedirectly marked (flagged, coded, etc.) to include the potentialactionable dental features. Thus, the data structure may be acompilation of all of the different records. The combined/compiled datastructure may be referred to as a marked data structure or an actionabledental feature data structure.

Any of the records, including the near-IR 2D images, may be used/scannedto identify the potential actionable dental features. As described abovewhen a suspicious area is identified, either automatically,semi-automatically/semi-manually, or manually (e.g., by a user), in oneof the records, the method or system may then search the correspondingarea of the dental arch on all or some of the other records and concludeif there is a finding. In some variations, the method or apparatus mayupdate the images on all or some of the records (and/or in the combineddata structure) based on the analysis described herein.

In any of the methods and systems described herein, tooth segmentationmay be used on all or some of the records and/or the 3D model to enhanceperformance and usability. Tooth segmentation may be added prior tovolumetric modeling to assist and improve volumetric results and modelquality. For example, the volumetric 3D model may uses the informationof segmentation to potentially enhance performance as additional surface3D information is added. The segmentation information may also assist insegmenting enamel-dentin-lesions to improve auto detection andsuspicious areas marking (e.g., including but not limited to when usingan automatic agent to identify potential actionable dental features).Alternatively or additionally, tooth segmentation may be added to thevolumetric modeling post-processing to assist in segmentingenamel-dentin-lesions to improve auto detection and suspicious areasmarking. For example, segmentation may also or alternatively help withcorrelating the structures between different imaging modalities,including registering findings on volumetric with other modalities toprovide cross-modality visualization. Tooth segmentation may be used toimprove records and cross-modality visualization of clinical findingsand annotations

In any of the methods and apparatuses described herein the confidencelevel indicated may be a quantitative and/or qualitative index. Forexample, a quantitative confidence level “score” may be provided (e.g.,using a number between, for example, 0-100, 0 to 1.0, −100 to 100, orscaled to any range of numeric values). Qualitative indexes may include“high, medium high, medium, medium low, low”, etc. Both qualitative andquantitative confidence levels may be used. A rating system for theconfidence level based on the multiple records as described herein maybe impactful for insurance claims and/or patient communication.

In any of the methods and system described herein, the morphology of thedental arch may be used to help identify the likely areas of interest orpotential issues. Thus, in general, the 3D model (volumetric model) maybe used and/or modified as described herein in order to include theregions of potential actionable dental features. A modified 3D model mayact as a map that visually indicates areas of areas for risk assessment;this may be used, for example, to guide treatment of the patient,including to promote use of sealants, orthodontic treatment or nightguards, etc. In some variations, the modified 3D model may be used toguide a user when additional scans are needed (e.g., when there is a lownumber of scans in the risk areas). As used herein, a modified 3D modelmay include a 3D (e.g., volumetric and/or surface) model that has beenmarked to indicate the locations and/or type and/or confidence level ofpotential actionable dental features. Thus, in general, the use ofadditional data sources to guide users to capture potential areas ofinterest (e.g., when they appear in records, and particularly in recordsother than near-IR/NIRI scan) may help confirm findings of potentialactionable dental features. As mentioned, the results, including amodified 3D model, may help guide the user in scanning or re-scanning(at a future time) the user's dentition. For example, historical scanscan be used as a targeting map while scanning (and to confirm adequatecoverage in those areas). One or more derived images/presentations maybe used in addition or alternatively. For example, tooth segmentationmay be used to generate a tooth chart map (e.g., from the 3D volumetricmodel) that can be used for follow up and auto import into dentalpractice management software (DPMS). For example, individual records maybe lined to match a specified problem to a tooth map.

As described above, an intraoral scanning system may include a hand-heldwand configured to operate with one or more sensors to detect infraredand visible light, a sleeve configure to be placed over a distal end ofthe hand-held wand having a transparent window at a distal end region,and one or more processors operably connected to the hand-held wand, theone or more processors configured to: receive visible light informationand infrared information from the one or more sensors; determine, inreal time, surface information from the visible light information andgenerate a three-dimensional (3D) surface model of a subject's teethusing the surface information; display the 3D surface model on a displayscreen in real time as the hand-held wand is moved; determine, in realtime, projections into the subject's teeth from the infraredinformation, and generate one or more images into the subject's teethfrom the projections; and display, in real time on the display screen,one or more images into the subject's teeth as the hand-held wand ismoved. As described above, the surface information may be processedand/or displayed concurrently with or separately from the infraredinformation. For example, in variations in which different sleeves areused for surface and infrared (e.g., transillumination) the images intothe subject's teeth may be displayed (in real time) when scanning withthe infrared (e.g., near-IR) sleeve on the wand, while the surfaceimages may be displayed with the surface scanning sleeve on the wand. Insome variations, the surface and near-IR may be scanned concurrently oralternately (e.g., see FIG. 8, above). For example, a sleeve may beconfigured for operating with both an infrared light source and avisible light source.

The one or more processors may be configured to: display a viewingwindow over at least a portion the 3D surface model, receive, from auser, a change in a relative position between the viewing window and the3D surface model, identify, from both the 3D surface model and aplurality of images of the subject's teeth taken from different anglesand positions relative to the subject's teeth, a near-infrared (near-IR)image taken at an angle and position that approximates a relative angleand position between the viewing window relative to the 3D surfacemodel, and display the identified near-IR image taken at the angle andposition that approximates the angle and position between the viewingwindow relative to the 3D surface model.

For example, an intraoral scanning system may include: a hand-held wandconfigured to operate in a plurality of imaging modes, including anear-infrared (near-IR) imaging mode and a visible light imaging mode; asleeve configure to be placed on a distal end of the hand-held wand; andone or more processors operably connected to the hand-held wand, the oneor more processors configured to: receive visible light informationduring a visible light imaging mode and near-IR light information duringa near-IR imaging mode; determine, in real time, surface informationfrom the visible light information and generate a three-dimensional (3D)surface model of a subject's teeth using the surface information;display the 3D surface model on a display screen in real time as thehand-held wand is moved; determine, in real time, projections into thesubject's teeth from the near-IR light information, and generate imagesinto the subject's teeth from the projections; and display, in real timeon the display screen, the images into the subject's teeth as thehand-held wand is moved.

For example, as described and illustrated above, an intraoral scanningsystem may include: a hand-held wand configured to operate in aplurality of imaging modes, including a near-infrared (near-IR) imagingmode and a visible light imaging mode; a sleeve configure to be placedon a distal end of the hand-held wand having a transparent window at adistal end region, wherein the sleeve comprises a pair of wingsextending on either side of the transparent window, and a near-IR lightsource in at least one of the wings configured to project near-IR lightbetween the wings; and one or more processors operably connected to thehand-held wand, the one or more processors configured to: receivevisible light information during a visible light imaging mode andnear-IR light information during a near-IR imaging mode; determine, inreal time, surface information from the visible light information andgenerate a three-dimensional (3D) surface model of a subject's teethusing the surface information; display the 3D surface model on a displayscreen in real time as the hand-held wand is moved; determine, in realtime, projections into the subject's teeth from the near-IR lightinformation, and generate images into the subject's teeth from theprojections; and display, in real time on the display screen, the imagesinto the subject's teeth as the hand-held wand is moved.

Any of the methods (including user interfaces) described herein may beimplemented as software, hardware or firmware, and may be described as anon-transitory computer-readable storage medium storing a set ofinstructions capable of being executed by a processor (e.g., computer,tablet, smartphone, etc.), that when executed by the processor causesthe processor to control perform any of the steps, including but notlimited to: displaying, communicating with the user, analyzing,modifying parameters (including timing, frequency, intensity, etc.),determining, alerting, or the like.

When a feature or element is herein referred to as being “on” anotherfeature or element, it can be directly on the other feature or elementor intervening features and/or elements may also be present. Incontrast, when a feature or element is referred to as being “directlyon” another feature or element, there are no intervening features orelements present. It will also be understood that, when a feature orelement is referred to as being “connected”, “attached” or “coupled” toanother feature or element, it can be directly connected, attached orcoupled to the other feature or element or intervening features orelements may be present. In contrast, when a feature or element isreferred to as being “directly connected”, “directly attached” or“directly coupled” to another feature or element, there are nointervening features or elements present. Although described or shownwith respect to one embodiment, the features and elements so describedor shown can apply to other embodiments. It will also be appreciated bythose of skill in the art that references to a structure or feature thatis disposed “adjacent” another feature may have portions that overlap orunderlie the adjacent feature.

Terminology used herein is for the purpose of describing particularembodiments only and is not intended to be limiting of the invention.For example, as used herein, the singular forms “a”, “an” and “the” areintended to include the plural forms as well, unless the context clearlyindicates otherwise. It will be further understood that the terms“comprises” and/or “comprising,” when used in this specification,specify the presence of stated features, steps, operations, elements,and/or components, but do not preclude the presence or addition of oneor more other features, steps, operations, elements, components, and/orgroups thereof. As used herein, the term “and/or” includes any and allcombinations of one or more of the associated listed items and may beabbreviated as “/”.

Spatially relative terms, such as “under”, “below”, “lower”, “over”,“upper” and the like, may be used herein for ease of description todescribe one element or feature's relationship to another element(s) orfeature(s) as illustrated in the figures. It will be understood that thespatially relative terms are intended to encompass differentorientations of the device in use or operation in addition to theorientation depicted in the figures. For example, if a device in thefigures is inverted, elements described as “under” or “beneath” otherelements or features would then be oriented “over” the other elements orfeatures. Thus, the exemplary term “under” can encompass both anorientation of over and under. The device may be otherwise oriented(rotated 90 degrees or at other orientations) and the spatially relativedescriptors used herein interpreted accordingly. Similarly, the terms“upwardly”, “downwardly”, “vertical”, “horizontal” and the like are usedherein for the purpose of explanation only unless specifically indicatedotherwise.

Although the terms “first” and “second” may be used herein to describevarious features/elements (including steps), these features/elementsshould not be limited by these terms, unless the context indicatesotherwise. These terms may be used to distinguish one feature/elementfrom another feature/element. Thus, a first feature/element discussedbelow could be termed a second feature/element, and similarly, a secondfeature/element discussed below could be termed a first feature/elementwithout departing from the teachings of the present invention.

Throughout this specification and the claims which follow, unless thecontext requires otherwise, the word “comprise”, and variations such as“comprises” and “comprising” means various components can be co-jointlyemployed in the methods and articles (e.g., compositions and apparatusesincluding device and methods). For example, the term “comprising” willbe understood to imply the inclusion of any stated elements or steps butnot the exclusion of any other elements or steps.

In general, any of the apparatuses and methods described herein shouldbe understood to be inclusive, but all or a sub-set of the componentsand/or steps may alternatively be exclusive, and may be expressed as“consisting of” or alternatively “consisting essentially of” the variouscomponents, steps, sub-components or sub-steps.

As used herein in the specification and claims, including as used in theexamples and unless otherwise expressly specified, all numbers may beread as if prefaced by the word “about” or “approximately,” even if theterm does not expressly appear. The phrase “about” or “approximately”may be used when describing magnitude and/or position to indicate thatthe value and/or position described is within a reasonable expectedrange of values and/or positions. For example, a numeric value may havea value that is +/−0.1% of the stated value (or range of values), +/−1%of the stated value (or range of values), +/−2% of the stated value (orrange of values), +/−5% of the stated value (or range of values), +/−10%of the stated value (or range of values), etc. Any numerical valuesgiven herein should also be understood to include about or approximatelythat value, unless the context indicates otherwise. For example, if thevalue “10” is disclosed, then “about 10” is also disclosed. Anynumerical range recited herein is intended to include all sub-rangessubsumed therein. It is also understood that when a value is disclosedthat “less than or equal to” the value, “greater than or equal to thevalue” and possible ranges between values are also disclosed, asappropriately understood by the skilled artisan. For example, if thevalue “X” is disclosed the “less than or equal to X” as well as “greaterthan or equal to X” (e.g., where X is a numerical value) is alsodisclosed. It is also understood that the throughout the application,data is provided in a number of different formats, and that this data,represents endpoints and starting points, and ranges for any combinationof the data points. For example, if a particular data point “10” and aparticular data point “15” are disclosed, it is understood that greaterthan, greater than or equal to, less than, less than or equal to, andequal to 10 and 15 are considered disclosed as well as between 10 and15. It is also understood that each unit between two particular unitsare also disclosed. For example, if 10 and 15 are disclosed, then 11,12, 13, and 14 are also disclosed.

Although various illustrative embodiments are described above, any of anumber of changes may be made to various embodiments without departingfrom the scope of the invention as described by the claims. For example,the order in which various described method steps are performed mayoften be changed in alternative embodiments, and in other alternativeembodiments one or more method steps may be skipped altogether. Optionalfeatures of various device and system embodiments may be included insome embodiments and not in others. Therefore, the foregoing descriptionis provided primarily for exemplary purposes and should not beinterpreted to limit the scope of the invention as it is set forth inthe claims.

The examples and illustrations included herein show, by way ofillustration and not of limitation, specific embodiments in which thesubject matter may be practiced. As mentioned, other embodiments may beutilized and derived there from, such that structural and logicalsubstitutions and changes may be made without departing from the scopeof this disclosure. Such embodiments of the inventive subject matter maybe referred to herein individually or collectively by the term“invention” merely for convenience and without intending to voluntarilylimit the scope of this application to any single invention or inventiveconcept, if more than one is, in fact, disclosed. Thus, althoughspecific embodiments have been illustrated and described herein, anyarrangement calculated to achieve the same purpose may be substitutedfor the specific embodiments shown. This disclosure is intended to coverany and all adaptations or variations of various embodiments.Combinations of the above embodiments, and other embodiments notspecifically described herein, will be apparent to those of skill in theart upon reviewing the above description.

What is claimed is:
 1. An intraoral scanning system, comprising: ahand-held wand configured to operate with one or more sensors to detectinfrared and visible light, wherein the one or more sensors comprises animage sensor; a sleeve configured to be placed over a distal end of thehand-held wand having a window at a distal end region; and one or moreprocessors operably connected to the hand-held wand, the one or moreprocessors configured to: receive visible light information and infraredinformation from the one or more sensors; determine, in real time,surface information from the visible light information and generate athree-dimensional (3D) surface model of a subject's teeth using thesurface information; display the 3D surface model on a display screen inreal time as the hand-held wand is moved; capture, in real time usingthe image sensor, one or more two-dimensional (2D) images of an internalregion of the subject's teeth from the infrared information; anddisplay, on the display screen, the one or more 2D images of theinternal region of the subject's teeth.
 2. The system of claim 1,wherein the sleeve comprises a pair of wings extending on either side ofthe window.
 3. The system of claim 2, wherein at least one of the wingshouses a near-infrared (near-IR) light source.
 4. The system of claim 2,wherein the hand-held wand is configured to receive near-IR light thatis trans-illuminated through the subject's teeth.
 5. The system of claim1, wherein the sleeve is configured to make electrical contact with thewand so that the wand provides power to one or more light sources on thesleeve.
 6. The system of claim 1, wherein the sleeve comprises a firstsleeve configured for use with a visible light source.
 7. The system ofclaim 6, further comprising a second sleeve comprising an infrared lightsource configured for trans-illumination.
 8. The system of claim 1,wherein the sleeve is configured for operating with both an infraredlight source and a visible light source.
 9. The system of claim 1,wherein the one or more processors are configured to receive infraredinformation that is near-infrared light information.
 10. The system ofclaim 1, wherein the one or more processors are configured toconcurrently display the 3D surface model and the one or more 2D imagesof the internal region of the subject's teeth.
 11. The system of claim1, wherein the one or more processors operably connected to thehand-held wand are further configured to: display a viewing window overat least a portion the 3D surface model; receive, from a user, a changein a relative position between the viewing window and the 3D surfacemodel; identify, from both the 3D surface model and a plurality ofimages of the subject's teeth taken from different angles and positionsrelative to the subject's teeth, a 2D image of the internal region ofthe subject's teeth comprising a near-infrared (near-IR) image taken atan angle and position that approximates a relative angle and positionbetween the viewing window relative to the 3D surface model; and displaythe identified near-IR image taken at the angle and position thatapproximates the angle and position between the viewing window relativeto the 3D surface model.
 12. An intraoral scanning system, comprising: ahand-held wand configured to operate in a plurality of imaging modes,including a near-infrared (near-IR) imaging mode and a visible lightimaging mode; a sleeve configured to be placed on a distal end of thehand-held wand; and one or more processors operably connected to thehand-held wand, the one or more processors configured to: receivevisible light information during a visible light imaging mode andnear-IR light information during a near-IR imaging mode; determine, inreal time, surface information from the visible light information andgenerate a three-dimensional (3D) surface model of a subject's teethusing the surface information; display the 3D surface model on a displayscreen in real time as the hand-held wand is moved; capture, in realtime using an image sensor, one or more two-dimensional (2D) images ofan internal region of the subject's teeth from the near-IR lightinformation; and display, on the display screen, the one or more 2Dimages of the internal region of the subject's teeth.
 13. The system ofclaim 12, wherein the sleeve comprises a pair of wings extending oneither side of a distal end of the sleeve.
 14. The system of claim 13,wherein at least one of the wings houses a near-IR light source.
 15. Thesystem of claim 13, wherein the hand-held wand is configured to receivenear-IR light that is trans-illuminated through the subject's teeth. 16.The system of claim 12, wherein the sleeve is configured to makeelectrical contact with the wand so that the wand provides power to oneor more light sources on the sleeve.
 17. The system of claim 12, whereinthe sleeve comprises a first sleeve configured for use with visiblelight.
 18. The system of claim 17, further comprising a second sleevecomprising a near-IR light source configured for trans-illumination. 19.The system of claim 12, wherein the sleeve is configured for operatingwith both a near-IR light source and a visible light source.
 20. Thesystem of claim 12, wherein the one or more processors are configured toconcurrently display the 3D surface model and the one or more 2D imagesof the internal region of the subject's teeth.
 21. The system of claim12, wherein the one or more processors are further configured to:display a viewing window over at least a portion the 3D surface model;receive, from a user, a change in a relative position between theviewing window and the 3D surface model; identify, from both the 3Dsurface model and a plurality of images of the subject's teeth takenfrom different angles and positions relative to the subject's teeth, a2D image of the internal region of the subject's teeth comprising anear-infrared (near-IR) image taken at an angle and position thatapproximates a relative angle and position between the viewing windowrelative to the 3D surface model; and display the identified near-IRimage taken at the angle and position that approximates the angle andposition between the viewing window relative to the 3D surface model.22. An intraoral scanning system, comprising: a hand-held wandconfigured to operate in a plurality of imaging modes, including anear-infrared (near-IR) imaging mode and a visible light imaging mode; asleeve configured to be placed on a distal end of the hand-held wandhaving a distal end region, wherein the sleeve comprises a pair of wingsextending from the distal end region, and a near-IR light source in atleast one of the wings configured to project trans-illuminating near-IRlight between the wings; and one or more processors operably connectedto the hand-held wand, the one or more processors configured to: receivevisible light information during a visible light imaging mode andnear-IR light information during a near-IR imaging mode; determine, inreal time, surface information from the visible light information andgenerate a three-dimensional (3D) surface model of a subject's teethusing the surface information; display the 3D surface model on a displayscreen in real time as the hand-held wand is moved; capture, in realtime, one or more two-dimensional (2D) images of an internal region thesubject's teeth from the trans-illuminating near-IR light; and display,on the display screen, the one or more 2D images of the internal regionof the subject's teeth.
 23. The system of claim 22, wherein the sleeveis configured to make electrical contact with the wand so that the wandprovides power to one or more light sources on the sleeve.
 24. Thesystem of claim 22, wherein the sleeve further comprises a circuitryconfigured to connect to the near-IR light source configured to controlthe near-IR light source for transillumination.
 25. The system of claim22, wherein the sleeve is configured to be held by an attachment to thedistal end of the wand.
 26. The system of claim 22, wherein the sleeveis configured to be held by friction to the wand.
 27. The system ofclaim 22, wherein the wand is configured to wirelessly communicate withthe one or more processors.
 28. The system of claim 22, furthercomprising a second sleeve configured for use with a visible lightsource.
 29. The system of claim 22, wherein the one or more processorsare configured to concurrently display the 3D surface model and the oneor more 2D images of the internal region of the subject's teeth.
 30. Thesystem of claim 22, wherein the one or more processors operablyconnected to the hand-held wand are further configured to: display aviewing window over at least a portion the 3D surface model; receive,from a user, a change in a relative position between the viewing windowand the 3D surface model; identify, from both the 3D surface model and aplurality of images of the subject's teeth taken from different anglesand positions relative to the subject's teeth, a 2D image of theinternal region of the subject's teeth comprising a near-infrared(near-IR) image taken at an angle and position that approximates arelative angle and position between the viewing window relative to the3D surface model; and display the identified near-IR image taken at theangle and position that approximates the angle and position between theviewing window relative to the 3D surface model.